#26 Allow python-dslib to bundle python-pyasn1 and python-suds
Closed: Invalid None Opened 13 years ago by ppisar.

= Proposal topic =

To allow to bundle modified python-pyasn1 and python-suds libraries in new package python-dslib.

= Overview =

python-dslib-1.1 package under review [https://bugzilla.redhat.com/show_bug.cgi?id=648898] bundles two modified libraries:

  • pythons-suds – dslib developers modified suds library, changes have been sent to upstream, upstream declared they will be included in new version.

  • pythons-pyasn1 – dslib developers modified pyasn1-0.0.9a library heavily to get certificate revocation list support and fulfill other their needs. Developers stated pyasn1 upstream is not willing to incorporate the changes.

= Problem space =

We are trying to deliver python-dslib library to Fedora and RHEL users. dslib is a Python implementation of client for ISDS ([http://www.datoveschranky.info/] (Czech)) interface, a message delivery system endorsed by Czech government (obligatory for courts, state and municipality offices, police, firms, self-employees etc.; facultative for private persons).

= Solution Overview =

We will bundle suds library until suds upstream will release new version.

We will bundle pyasn1 library forever.

= Active Ingredients =

Package owner, Security team.

= Owners =

ppisar, a packager.


PS: This is copy of [https://fedorahosted.org/fesco/ticket/484] that has been closed due to competency changes of FESCo.

PPS: Proper component for this request is missing in FPC Trac.


Now that FPC has taken over approval of exceptions, we've put together some standard questions that should be answered:

https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries#Standard_questions

There's also some guidance on possible reasons that an exception could be granted. In addition, I've asked FPC to consider this as an additional reason: https://fedorahosted.org/fpc/ticket/33 which bears on the python-suds case.

pyasn1 needs more information. With the information given, it's not likely to be granted an exception.

once you've answered the standard questions for both bundled libraries, add Meeting to the trac keywords and we'll decide whether either or both libraries can be granted an exception.

Regarding pyasn1 library:

  • Has the library behaviour been modified?

Yes.

  • If so, how has it been modified?

Schemata for certificate revocation list added. Function to process such ASN.1 blobs added. Useless functions removed to speed up by skipping unneeded parsing.

  • Why haven't the changes been pushed to the upstream library?

Upstream does not like the changes. Upstream is two next version away.

  • Have the changes been proposed to the Fedora package maintainer for the library?

No, because they are not compatible with newer versions.

  • Could we make the forked version the canonical version within Fedora?

No. Upstream is not death and upstream versions are incompatible.

  • Are the changes useful to consumers other than the bundling application?

Maybe as current upstream library does not support CRL.

  • Is upstream keeping the base library updated or are they continuously one or more versions behind the latest upstream release?

The library is frozen probably.

  • What is the attitude of upstream towards bundling? (Are they eager to remove the bundled version? are they engaged with the upstream for the library? Do they have a history of bundling? Are they argumentative?)

Incompatible changes.

  • Overview of the security ramifications of bundling

Python library for parsing ASN.1 used to decapsulate cryptographic material.

  • Does the maintainer of the Fedora package of the library being bundled have any comments about this?

I don't know. Actually he lacks two version behind upstream (1.5 year).

  • Is there a plan for unbundling the library at a later time?

No.

  • Please include any relevant documentation -- mailing list links, bug reports for upstream or the bundled library, etc.

My e-mail correspondence with dslib, bundling, upstream.

This is diff against 0.0.9a and dslib' pyasn1:

{{{
diff -aurw /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/codec/ber/decoder.py ./codec/ber/decoder.py
--- /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/codec/ber/decoder.py 2010-11-22 10:47:11.467525531 +0100
+++ ./codec/ber/decoder.py 2010-09-16 08:44:58.000000000 +0200
@@ -33,7 +33,7 @@
state, decodeFun):
if not substrate:
raise error.PyAsn1Error('Empty substrate')
- octets = map(ord, substrate)
+ octets = map(ord, str(substrate))
if octets[0] & 0x80:
value = -1L
else:
@@ -101,6 +101,17 @@
)
return r, substrate

+class RevCertsListDecoder(AbstractDecoder):
+ '''
+ Decoder for RevCertsList - does not decode the content- returns
+ it as one string.
+ '''
+ protoComponent = univ.RevCertsList()
+ def valueDecoder(self, substrate, asn1Spec, tagSet, length,
+ state, decodeFun):
+ r = self._createComponent(tagSet, asn1Spec).clone(substrate)
+ return r, substrate
+
class OctetStringDecoder(AbstractDecoder):
protoComponent = univ.OctetString('')
def valueDecoder(self, substrate, asn1Spec, tagSet, length,
@@ -230,6 +241,30 @@
return r, substrate
component, substrate = decodeFun(substrate, asn1Spec)
if component == eoo.endOfOctets:
+ '''
+ Needed to add action when the explicit tag
+ has indefinite length. We need to close the
+ component with double EOC, not just one!
+ We set the property of tag, that it has indefinite length
+ and we check this property when we are completing
+ the component. If the explicit tag had the indefinite length,
+ we must eat one more EOC from substrate!
+ '''
+ #print '------'
+ # we can have more explicit tags, each with indef length
+ expectedExplicitEocCounter = 0
+ for tag in tagSet:
+ # print tag.length
+ if tag.length == -1:
+ expectedExplicitEocCounter += 1
+ #decrease by one - one tag is base tag (eoc solved by decoder), others are explicit
+ expectedExplicitEocCounter -= 1
+ #print '------'
+ if expectedExplicitEocCounter:
+ # NOTE: asn1Spec is None, as we are 'decoding' EOC
+ for i in xrange(expectedExplicitEocCounter):
+ spare_eoc, substrate = decodeFun(substrate, None)
+ #print spare_eoc
break
idx = self._getPositionByType(r, component, idx)
r.setComponentByPosition(idx, component)
@@ -312,29 +347,6 @@
class UTCTimeDecoder(OctetStringDecoder):
protoComponent = useful.UTCTime()

-class AnyDecoder(ChoiceDecoder):
- protoComponent = univ.Any
- def init(self, header):
- self.__header = header
- def _createComponent(self, tagSet, asn1Spec):
- if asn1Spec is None:
- return self.protoComponent(tagSet=tagSet)
- else:
- return asn1Spec.clone()
- def valueDecoder(self, substrate, asn1Spec, tagSet,
- length, state, decodeFun):
- if not decodeFun:
- return r, substrate
- component, new_substrate = decodeFun(
- substrate, None, tagSet, length, state
- )
- assert substrate.endswith(new_substrate)
- if new_substrate:
- substrate = substrate[:-len(new_substrate)]
- return univ.Any(self.__header+substrate), new_substrate
-
- indefLenValueDecoder = valueDecoder
-
codecMap = {
eoo.endOfOctets.tagSet: EndOfOctetsDecoder(),
univ.Integer.tagSet: IntegerDecoder(),
@@ -361,7 +373,8 @@
char.BMPString.tagSet: BMPStringDecoder(),
# useful types
useful.GeneralizedTime.tagSet: GeneralizedTimeDecoder(),
- useful.UTCTime.tagSet: UTCTimeDecoder()
+ useful.UTCTime.tagSet: UTCTimeDecoder(),
+
}

( stDecodeTag, stDecodeLength, stGetValueDecoder, stGetValueDecoderByAsn1Spec,
@@ -370,12 +383,12 @@
class Decoder:
def init(self, codecMap):
self.codecMap = codecMap
+ self.indefLen = False
def __call
(self, substrate, asn1Spec=None, tagSet=None,
length=None, state=stDecodeTag, recursiveFlag=1):
# Decode tag & length
while state != stStop:
if state == stDecodeTag:
- substrate_full = substrate
# Decode tag
if not substrate:
raise error.SubstrateUnderrunError(
@@ -416,6 +429,8 @@
if firstOctet == 128:
size = 1
length = -1
+ # set length of the last added tag to -1 (tags are added to the first place)
+ tagSet[0].length = -1
elif firstOctet < 128:
length, size = firstOctet, 1
else:
@@ -439,7 +454,6 @@
raise error.SubstrateUnderrunError(
'%d-octet short' % (length - len(substrate))
)
- substrate_header = substrate_full[:-len(substrate) or None]
if state == stGetValueDecoder:
if asn1Spec is None:
state = stGetValueDecoderByTag
@@ -482,12 +496,7 @@
__chosenSpec = asn1Spec
else:
__chosenSpec = None
- if __chosenSpec is None and isinstance(asn1Spec, dict) and \
- isinstance(asn1Spec.get(univ.Any.tagSet), univ.Any):
- concreteDecoder = AnyDecoder(substrate_header)
- asn1Spec = None
- state = stDecodeValue
- elif __chosenSpec is None or not\
+ if __chosenSpec is None or not\
__chosenSpec.getTypeMap().has_key(tagSet):
state = stTryAsExplicitTag
else:
@@ -497,6 +506,10 @@
baseTagSet = tag.TagSet(baseTag, baseTag)
else:
baseTagSet = tag.TagSet()
+ # chceck if we are decoding special case - RevCertsList
+ if hasattr(__chosenSpec, univ.REV_CERT_LIST_IDENTIFIER):
+ concreteDecoder = RevCertsListDecoder()
+ else:
concreteDecoder = self.__codecMap.get( # tagged subtype
baseTagSet
)
@@ -512,6 +525,9 @@
# Assume explicit tagging
state = stDecodeTag
else:
+ #f = open("error", "w")
+ #f.write(substrate)
+ #f.close()
raise error.PyAsn1Error(
'%s not in asn1Spec: %s' % (tagSet, asn1Spec)
)
@@ -526,6 +542,7 @@
stGetValueDecoder, decodeFun
)
else:
+
value, _substrate = concreteDecoder.valueDecoder(
substrate[:length], asn1Spec, tagSet,
length, stGetValueDecoder, decodeFun
diff -aurw /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/codec/ber/encoder.py ./codec/ber/encoder.py
--- /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/codec/ber/encoder.py 2010-11-22 10:47:11.467525531 +0100
+++ ./codec/ber/encoder.py 2010-07-09 09:06:33.000000000 +0200
@@ -178,6 +178,10 @@

class SequenceOfEncoder(AbstractItemEncoder):
def _encodeValue(self, encodeFun, value, defMode, maxChunkSize):
+ # check if we are encoding REV_CERT_LIST
+ if hasattr(value, univ.REV_CERT_LIST_IDENTIFIER):
+ substrate = value._value
+ return substrate, 1
if hasattr(value, 'setDefaultComponents'):
value.setDefaultComponents()
value.verifySizeSpec()
@@ -194,16 +198,6 @@
) + substrate
return substrate, 1

-class AnyEncoder(AbstractItemEncoder):
- def encodeTag(self, t, isConstructed):
- if isConstructed:
- return chr(t[0]|t[1]|t[2]|tag.tagFormatConstructed)
- else:
- return chr(t[0]|t[1]|t[2])
- def _encodeValue(self, encodeFun, value, defMode, maxChunkSize):
- assert len(value._value) <= maxChunkSize
- return str(value._value), 0
-
codecMap = {
eoo.endOfOctets.tagSet: EndOfOctetsEncoder(),
univ.Boolean.tagSet: IntegerEncoder(),
@@ -244,9 +238,6 @@
if len(tagSet) > 1:
concreteEncoder = explicitlyTaggedItemEncoder
else:
- if isinstance(value, univ.Any):
- concreteEncoder = AnyEncoder()
- else:
concreteEncoder = self.codecMap.get(tagSet)
if not concreteEncoder:
# XXX
diff -aurw /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/__init
.py ./init.py
--- /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/init.py 2005-06-14 08:59:38.000000000 +0200
+++ ./init.py 2010-07-09 09:06:33.000000000 +0200
@@ -1 +1,30 @@
majorVersionId = '1'
+'''
+import os
+import sys
+import string
+
+def switchApiVersion(subPkg):
+ pkg = os.path.split(path[0])[-1]
+ newMod = import(subPkg, globals(), locals(), pkg)
+ realPkg = '_real
' + pkg
+ if sys.modules.has_key(realPkg):
+ sys.modules[pkg] = sys.modules[realPkg]
+ sys.modules[realPkg] = sys.modules[pkg]
+ sys.modules[pkg] = newMod
+
+def isSubPackage(subDir):
+ if subDir and subDir[0] == 'v' and subDir[1] in string.digits \
+ and len(subDir) == 2:
+ return 1
+
+subDirs = filter(__isSubPackage, os.listdir(__path
[0]))
+subDirs.sort(); subDirs.reverse()
+
+if os.environ.has_key('PYASN1_API_VERSION'):
+ v = os.environ['PYASN1_API_VERSION']
+ if v:
+ switchApiVersion(v) # do not load any API
+else:
+ switchApiVersion(subDirs[-1]) # take the most recent version
+'''
\ Chybí znak konce řádku na konci souboru
Pouze v .: LICENSE
diff -aurw /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/type/base.py ./type/base.py
--- /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/type/base.py 2006-05-11 21:43:01.000000000 +0200
+++ ./type/base.py 2010-07-09 09:06:33.000000000 +0200
@@ -45,9 +45,10 @@
return self._tagSet.isSuperTagSetOf(other.getTagSet()) and \
self._subtypeSpec.isSuperTypeOf(other.getSubtypeSpec())

-class NoValue:
+class __NoValue(object):
def __getattr
(self, attr):
- raise error.PyAsn1Error('No value for %s()' % attr)
+ raise AttributeError('No value for %s()' % attr)
+ #raise error.PyAsn1Error('No value for %s()' % attr)
noValue = __NoValue()

# Base class for "simple" ASN.1 objects. These are immutable.
diff -aurw /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/type/tag.py ./type/tag.py
--- /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/type/tag.py 2007-08-14 12:03:21.000000000 +0200
+++ ./type/tag.py 2010-07-09 09:06:33.000000000 +0200
@@ -21,7 +21,7 @@
tagCategoryUntagged = 0x04

class Tag:
- def init(self, tagClass, tagFormat, tagId):
+ def init(self, tagClass, tagFormat, tagId, length=0):
if tagId < 0:
raise error.PyAsn1Error(
'Negative tag ID (%s) not allowed' % tagId
@@ -29,6 +29,7 @@
self.tag = (tagClass, tagFormat, tagId)
self.__uniqTag = (tagClass, tagId)
self.__hashedUniqTag = hash(self.__uniqTag)
+ self.length = length
def __repr
(self):
return '%s(tagClass=%s, tagFormat=%s, tagId=%s)' % (
(self.class.name,) + self.__tag
diff -aurw /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/type/univ.py ./type/univ.py
--- /home/petr/fedora/python-pyasn1/pyasn1-0.0.9a/pyasn1/v1/type/univ.py 2010-11-22 10:47:11.473525591 +0100
+++ ./type/univ.py 2010-07-09 09:06:33.000000000 +0200
@@ -5,6 +5,24 @@
from pyasn1.type import base, tag, constraint, namedtype, namedval
from pyasn1 import error

+
+REV_CERT_LIST_IDENTIFIER = "rev_cert_list"
+
+class RevCertsList(base.AbstractSimpleAsn1Item):
+ tagSet = tag.initTagSet(
+ tag.Tag(tag.tagClassUniversal, tag.tagFormatConstructed, 0x10)
+ )
+
+ def init(self, value=None, tagSet=None, subtypeSpec=None,
+ namedValues=None):
+ # set attribute for idetification - value does not matter
+ # important is, that this attribute exists
+ setattr(self, REV_CERT_LIST_IDENTIFIER, True)
+ base.AbstractSimpleAsn1Item.init(
+ self, value, tagSet, subtypeSpec
+ )
+ #its_me = True
+
# "Simple" ASN.1 types (yet incomplete)

class Integer(base.AbstractSimpleAsn1Item):
@@ -223,6 +241,23 @@
def prettyOut(self, value):
return '\'%s\'B' % string.join(map(str, value), '')

  • def _tuple_to_byte(self, tuple):
  • s = ''.join(map(str, tuple))
  • return chr(int(s,2))
  • '''
  • Converts bit string into octets string
  • '''
  • def toOctets(self):
  • res = ''
  • byte_len = len(self._value) / 8
  • for byte_idx in xrange(byte_len):
  • bit_idx = byte_idx * 8
  • byte_tuple = self._value[bit_idx:bit_idx + 8]
  • byte = self._tuple_to_byte(byte_tuple)
  • res += byte
  • return res
  • class OctetString(base.AbstractSimpleAsn1Item):
    tagSet = tag.initTagSet(
    tag.Tag(tag.tagClassUniversal, tag.tagFormatSimple, 0x04)
    @@ -317,6 +352,9 @@
    r[-1][-1] = r[-1][:-1]
    return string.join(r, '.')

  • def str(self):

  • return self.prettyOut(self._value)
  • class Real(base.AbstractSimpleAsn1Item):
    tagSet = tag.initTagSet(
    tag.Tag(tag.tagClassUniversal, tag.tagFormatSimple, 0x09)
    @@ -393,6 +431,16 @@
    tag.Tag(tag.tagClassUniversal, tag.tagFormatConstructed, 0x10)
    )

+class StructuredOctetString(SetOf):
+ #componentType = OctetString()
+ tagSet = tag.initTagSet(
+ tag.Tag(tag.tagClassUniversal, tag.tagFormatConstructed, 0x04)
+ )
+
+ def add(self, value):
+ self._componentValues.append(value)
+ return self
+
class SequenceAndSetBase(base.AbstractConstructedAsn1Item):
componentType = namedtype.NamedTypes()
def _cloneComponentValues(self, myClone, cloneValueFlag):
@@ -629,10 +677,5 @@

 def setDefaultComponents(self): pass

-class Any(base.AbstractSimpleAsn1Item):
- tagSet = tag.TagSet() # untagged, XXX as in Choice
- defaultValue = ''
- def prettyOut(self, value): return repr(value)
-
# XXX
# coercion rules?
}}}

Replying to [comment:2 ppisar]:

Regarding pyasn1 library:
[...]
* Why haven't the changes been pushed to the upstream library?

Upstream does not like the changes. Upstream is two next version away.

I believe upstream might be more receptive to a patch which simply adds the missing features without deleting anything. Do you have a link to e-mail exchange with pyasn1 upstream?

  • Have the changes been proposed to the Fedora package maintainer for the library?

No, because they are not compatible with newer versions.

In what way are they not compatible?

[...]

  • Please include any relevant documentation -- mailing list links, bug reports for upstream or the bundled library, etc.

My e-mail correspondence with dslib, bundling, upstream.

Missing?

This is diff against 0.0.9a and dslib' pyasn1:

This doesn't look too bad to me.

Replying to [comment:3 rathann]:

I believe upstream might be more receptive to a patch which simply adds the missing features without deleting anything. Do you have a link to e-mail exchange with pyasn1 upstream?

I have not talked to pyasn1 upstream. Because:

In what way are they not compatible?

Upstream is far away, It uses the removed code.

I will contact pyasn1 upstream and propose the changes.

  • Please include any relevant documentation -- mailing list links, bug reports for upstream or the bundled library, etc.

My e-mail correspondence with dslib, bundling, upstream.

Missing?

I assume you are not interested in mails written in Czech ;)

But I have a good news: I talked to dslib developers again and they said they will try to resolve it.

Do did anything ever happen here? The previous comment indicates that upstream was working on resolving this, but we've had no further updates. The review ticket is still open, though it hasn't been touched this year.

Upstream achieved to merge their changes into pyasn1-0.0.13b (reflected in dslib-1.5). The suds issue has not been resolved yet. [https://git.nic.cz/redmine/issues/285#note-4 According upstream] the suds upstream is dead and thus they are not going to merge it.

When I find a spare time, I will repackage latest dslib and try to resolve this issue (by pushing suds upstream or forking suds).

The suds library has been forked and package separately.

The dslib package has been reviewed independently after that as [https://bugzilla.redhat.com/show_bug.cgi?id=854690 bug 854690].

This exception is not needed anymore. Thank you for your time.

Login to comment on this ticket.

Metadata