#21 Migrate to modernpaste
Closed: Invalid 4 years ago by ankursinha. Opened 7 years ago by ankursinha.

The fedora fpaste server is going to move from sticky-notes to modernpaste. We need to begin working on moving fpaste to use the modernpaste API.


Welp, I tried commenting, but it seems my comment got lost. Let's try again.

I have a diff that got this working as a proof-of-concept, but it's pretty hacky and would need cleaned up:

diff --git a/fpaste b/fpaste
index 7a2eefe..d33461a 100755
--- a/fpaste
+++ b/fpaste
@@ -20,7 +20,7 @@ VERSION = '0.3.8.1'
 USER_AGENT = 'fpaste/' + VERSION
 SET_DESCRIPTION_IF_EMPTY = 1  # stdin, clipboard, sysinfo
 # FPASTE_URL = 'http://fpaste.org/'
-FPASTE_URL = 'http://paste.fedoraproject.org/'
+FPASTE_URL = 'http://modernpaste.fedorainfracloud.org/api/paste'

 import os
 import sys
@@ -28,6 +28,7 @@ import urllib.request
 import urllib.parse
 import urllib.error
 import subprocess
+import time
 import json
 from optparse import OptionParser, OptionGroup, SUPPRESS_HELP

@@ -121,12 +122,11 @@ def paste(text, options):
     if len(author) > 50:
         author = author[0:50-3] + "..."

-    params = urllib.parse.urlencode(
-        {'paste_lang': options.lang, 'paste_data': text, 'paste_private':
-         options.make_private, 'paste_expire': options.expires,
-         'paste_password': options.password, 'paste_user': author, 'api_submit'
-         : 'true', 'mode': 'json'})
-    pasteSizeKiB = len(params)/1024.0
+    data = json.dumps(
+        {'language': options.lang, 'contents': text,
+         'expiry_time': int(options.expires) + int(time.time()),
+         'password': options.password}).encode('utf8')
+    pasteSizeKiB = len(data)/1024.0

     # 512KiB appears to be the current hard limit (20110404); old limit was
     # 16MiB
@@ -144,10 +144,11 @@ def paste(text, options):
             return [False, False]

     req = urllib.request.Request(
-        url=options.url,
-        data=params.encode(),
+        url=options.url + '/submit',
+        data=data,
         headers={
-            'User-agent': USER_AGENT})
+            'User-agent': USER_AGENT,
+            'Content-Type': 'application/json'})
     if options.proxy:
         if options.debug:
             print("Using proxy: %s" % options.proxy, file=sys.stderr)
@@ -174,42 +175,14 @@ def paste(text, options):
             file=sys.stderr)
         return [False, False]

-    if 'error' in response['result']:
-        error = response['result']['error']
-        if error == 'err_spamguard_php':
-            print(
-                "Error: Poster's IP address is listed as malicious",
-                file=sys.stderr)
-        elif error == 'err_spamguard_noflood':
-            print("Error: Poster is trying to flood", file=sys.stderr)
-        elif error == 'err_spamguard_stealth':
-            print(
-                "Error: The paste triggered the spam filter",
-                file=sys.stderr)
-        elif error == 'err_spamguard_ipban':
-            print("Error: Poster's IP address is banned", file=sys.stderr)
-        elif error == 'err_author_numeric':
-            print(
-                "Error: Poster's alias should be alphanumeric",
-                file=sys.stderr)
-        else:
-            print("Error: %s" % error, file=sys.stderr)
-        sys.exit(-1)
-
-    id = [i[1]["id"] for i in response.items()].pop()
-#        for k,j in i.iteritems():
-#            print j, k
-    if options.make_private == 'yes':
-        private_hash = [i[1]["hash"] for i in response.items()].pop()
-        url = "{0}{1}/{2}".format(options.url, id, private_hash)
-    else:
-        url = "{0}{1}".format(options.url, id)
+    url = response['url']

-    short_url = get_shortened_url(url, options.password)
-    if short_url:
-        return [url, short_url]
-    else:
-        return [url, False]
+    #short_url = get_shortened_url(url, options.password)
+    #if short_url:
+    #    return [url, short_url]
+    #else:
+    #    return [url, False]
+    return [url, False]


 def sysinfo(
@@ -565,7 +538,6 @@ def main():
                        "z80",
                        "zxbasic"]
     validClipboardSelectionOpts = ['primary', 'secondary', 'clipboard']
-    validPrivateOpts = ['yes', 'no']
     ext2lang_map = {
         'sh': 'bash',
         'bash': 'bash',
@@ -635,13 +607,6 @@ Examples:
         ', '.join(validExpiresOpts),
         metavar='EXPIRES')
     fpasteOrg_group.add_option(
-        '-P',
-        '--private',
-        help='make paste private; default is %default; valid options: ' +
-        ', '.join(validPrivateOpts),
-        dest='make_private',
-        metavar='"PRIVATE"')
-    fpasteOrg_group.add_option(
         '-U',
         '--URL',
         help='URL of fpaste server; default is %default',
@@ -725,10 +690,9 @@ Examples:
     parser.set_defaults(
         nick='',
         lang='text',
-        make_private='yes',
         expires='2592000',
         selection='primary',
-        password='',
+        password=None,
         url=FPASTE_URL)
     (options, args) = parser.parse_args()

I also talked with @stickster about the upstream CLI (https://github.com/LINKIWI/modern-paste-pb) - I'm not sure if we want to use any of that code or contribute anything back to them or whatnot. Upstream has been very responsive to our requests, so it would be nice to give back wherever we can.

@pingou did we lose the ability to write fenced code blocks at some point?

@ankursinha https://gist.github.com/relrod/f73cf436ecb1a33d8ac5ba8f4411e643 is the diff that does not seem to want to render correctly above.

Shouldn't be

  def foo(bar):
    print bar * 10

I wonder what the best way forward is - the upstream cli doesn't seem to have all the options that fpaste currently has, so we can't really use it as is. Also, there are a few options that will be fpaste specific, like --sysinfo, so it does look like we'll have to have a separate fpaste client after all - otherwise we could've just added the required features to upstream's client and used that.

At the moment, I'll refactor fpaste to use modernpaste based on your patch etc., and at the same time, I'll try sending the features upstream so that the upstream tool also benefits.

Does that sound OK @codeblock?

Sounds good to me. Let me know if you need any help/someone to test anything. :)

@codeblock - I've been working on this. Do peek at the modernpaste branch from time to time. I was updated the language list in this commit but noticed that neither our instance nor the upstream test site list all in the list here in their drop down boxes. Is this a config option for the server? Can we support them all? (We'll need to keep fpaste and the server config in sync otherwise).

General observation: it appears as if modernpaste supports less features (no nick, less languages, no shortened URL) than the stickynotes server.

@ankursinha I will take a look at the branch. Yeah, we can support them all, just need to add them to the server config. URL shortening is something I've added but have yet to send upstream. Lack of nick is something we can probably fix. We will eventually have optional authentication with FAS/Ipsilon (/cc @puiterwijk), but I agree it would be nice to let unauthenticated users supply a nickname. I'll ask upstream if they have any interest in carrying that change.

What else is needed -- apart from modernpaste deployment :wink: -- for an updated CLI release? I think @codeblock is going to check with the infra guys on a rollout timeline. So it would be good to have that whole plan, crossover time and user expectations understood.

fpaste on the modernpaste branch should be functional enough. I'd applied @codeblock 's patch and made some tweaks to mirror the modernpaste feature set (5 months ago according to the commit log!) but some things may be still missing (as the comments here show).

@codeblock would you know what the URL for the staging instance is please? (http://modernpaste.fedorainfracloud.org/api/paste isn't up any more)

Updated the script. Checking functionalities now.

So, what works:

  • title
  • language
  • password

What does not

  • expiry time: looking into this. The code seems correct, and it seems to be sending the right json too. @codeblock - any hints on this - can you maybe check server side? It always seems to set the expiry period to a week, irrespective of what expiry_time is sent:
[asinha@ankur  fpaste(modernpaste *=)]$ ./fpaste -x 1800
aqdsfasd
asdf
as
dfsa
Sending data b'{"expiry_time": 1482231398, "language": "text", "contents": "aqdsfasd\\nasdf\\nas\\ndfsa\\n", "password": null, "title": null}'
Uploading (0.1KiB)...
Received response: {'expiry_time': 1482834399, 'post_time': 1482229599, 'is_active': True, 'contents': 'aqdsfasd\nasdf\nas\ndfsa\n', 'language': 'text', 'paste_id_repr': '5UMxXmGk5TufDB7o32GRk8nRi-rYnlYzizqToCmG3BY=', 'is_password_protected': False, 'views': 0, 'attachments': [], 'title': 'Untitled', 'url': 'https://modernpaste.stg.fedoraproject.org/paste/5UMxXmGk5TufDB7o32GRk8nRi-rYnlYzizqToCmG3BY='}
https://modernpaste.stg.fedoraproject.org/paste/5UMxXmGk5TufDB7o32GRk8nRi-rYnlYzizqToCmG3BY=

[asinha@ankur  fpaste(modernpaste *=)]$ echo "1482834399 - 1482229599" | bc -l
604800
  • authentication/nick

Not sure how we're going to implement this tbh

I see we've migrated the server. Do you think the client is ready enough to make a release? Let me know and I can make one.

Got a feature request - also get deactivation link: https://pagure.io/fpaste/issue/33

So, what works:

expiry time: looking into this. The code seems correct, and it seems to be sending the right json too. @codeblock - any hints on this - can you maybe check server side? It always seems to set the expiry period to a week, irrespective of what expiry_time is sent:

This is exactly as intended.
We have said that without auth, pastes will always expire in a week.

[asinha@ankur fpaste(modernpaste *=)]$ ./fpaste -x 1800
aqdsfasd
asdf
as
dfsa
Sending data b'{"expiry_time": 1482231398, "language": "text", "contents": "aqdsfasd\nasdf\nas\ndfsa\n", "password": null, "title": null}'
Uploading (0.1KiB)...
Received response: {'expiry_time': 1482834399, 'post_time': 1482229599, 'is_active': True, 'contents': 'aqdsfasd\nasdf\nas\ndfsa\n', 'language': 'text', 'paste_id_repr': '5UMxXmGk5TufDB7o32GRk8nRi-rYnlYzizqToCmG3BY=', 'is_password_protected': False, 'views': 0, 'attachments': [], 'title': 'Untitled', 'url': 'https://modernpaste.stg.fedoraproject.org/paste/5UMxXmGk5TufDB7o32GRk8nRi-rYnlYzizqToCmG3BY='}
https://modernpaste.stg.fedoraproject.org/paste/5UMxXmGk5TufDB7o32GRk8nRi-rYnlYzizqToCmG3BY=

[asinha@ankur fpaste(modernpaste *=)]$ echo "1482834399 - 1482229599" | bc -l
604800

authentication/nick

Not sure how we're going to implement this tbh

OpenID Connect is what we will be using for this.

Thanks for the update @puiterwijk - for the time being, until we've figured out authentication, I can remove the expiry option and push an update. Does that sound OK?

Pushed an update to rawhide.

Someones's asked if the deactivation token can be obtained - the server doesn't return it in the json response. Is this waiting for authentication too?

Metadata Update from @ankursinha:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.

Metadata