Segfault when generating metadata on Arch

[antonio-rojas]

appstream-generator 0.6.2, compiled against ldc 1.1

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000590fff in asgen.result.GeneratorResult.finalize() (this=0x7f3a0bbbc100) at result.d:252
252     result.d: No existe el fichero o el directorio.
[Current thread is 1 (Thread 0x7f4a3e56d700 (LWP 1220))]
(gdb) thread apply all bt

Thread 8 (Thread 0x7f6a4bbce900 (LWP 1219)):
#0  0x00007f6a48cba48d in poll () at /usr/lib/libc.so.6
#1  0x00007f4a3c32e955 in  () at /usr/lib/libcurl.so
#2  0x00007f4a3c32934c in curl_multi_wait () at /usr/lib/libcurl.so
#3  0x00007f4a3c321c74 in curl_easy_perform () at /usr/lib/libcurl.so
#4  0x00007f6a497d258f in std.net.curl.Curl.perform(std.typecons.Flag!("throwOnError").Flag) ()
    at /usr/lib/libphobos2-ldc.so.71
#5  0x00007f6a497d241b in std.net.curl.HTTP.perform(std.typecons.Flag!("throwOnError").Flag) ()
    at /usr/lib/libphobos2-ldc.so.71
#6  0x000000000056385a in std.net.curl._basicHTTP!(ubyte)._basicHTTP(const(char)[], const(void)[], std.net.curl.HTTP) (url=..., sendData=..., client=...) at screenshothandler.d:1013
#7  0x0000000000562e28 in std.net.curl.get!(std.net.curl.HTTP, ubyte).get(const(char)[], std.net.curl.HTTP) (url=..., conn=...) at screenshothandler.d:536
#8  0x0000000000561be0 in asgen.handlers.screenshothandler.processScreenshot(asgen.result.GeneratorResult, appstream.Component.Component, appstream.Screenshot.Screenshot, immutable(char)[], uint) (gres=0x7f3a0bba5900, cpt=0x7f3a0bc4b980, scr=0x7f3a0bec2c80, mediaExportDir=..., scrNo=1) at screenshothandler.d:98
#9  0x0000000000561722 in asgen.handlers.screenshothandler.processScreenshots(asgen.result.GeneratorResult, appstream.Component.Component, immutable(char)[]) (gres=0x7f3a0bba5900, cpt=0x7f3a0bc4b980, mediaExportDir=...) at screenshothandler.d:53
#10 0x000000000052d777 in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7f4a35698500) at extractor.d:216
#11 0x00000000005188da in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
#12 0x000000000051db2c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
#13 0x00007f6a4972a32f in std.parallelism.submitAndExecute(std.parallelism.TaskPool, scope void() delegate) ()
    at /usr/lib/libphobos2-ldc.so.71
#14 0x0000000000518827 in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate) (this=..., dg=...) at /usr/include/dlang/ldc/std/parallelism.d:3835
#15 0x00000000005186dc in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler) (this=0x7f6a4bace900, pkgs=..., iconh=0x7f4a0ef4c480) at engine.d:134
#16 0x000000000051c079 in asgen.engine.Engine.run(immutable(char)[]) (this=0x7f6a4bace900, suite_name=...) at engine.d:505
#17 0x00000000004b95e8 in D main (args=...) at app.d:118

Thread 7 (Thread 0x7f4a37fff700 (LWP 1224)):
#0  0x00007f6a48cba48d in poll () at /usr/lib/libc.so.6
#1  0x00007f4a3c32e955 in  () at /usr/lib/libcurl.so
#2  0x00007f4a3c32934c in curl_multi_wait () at /usr/lib/libcurl.so
#3  0x00007f4a3c321c74 in curl_easy_perform () at /usr/lib/libcurl.so
#4  0x00007f6a497d258f in std.net.curl.Curl.perform(std.typecons.Flag!("throwOnError").Flag) ()
---Type <return> to continue, or q <return> to quit---
    at /usr/lib/libphobos2-ldc.so.71
#5  0x00007f6a497d241b in std.net.curl.HTTP.perform(std.typecons.Flag!("throwOnError").Flag) ()
    at /usr/lib/libphobos2-ldc.so.71
#6  0x000000000056385a in std.net.curl._basicHTTP!(ubyte)._basicHTTP(const(char)[], const(void)[], std.net.curl.HTTP) (url=..., sendData=..., client=...) at screenshothandler.d:1013
#7  0x0000000000562e28 in std.net.curl.get!(std.net.curl.HTTP, ubyte).get(const(char)[], std.net.curl.HTTP) (url=..., conn=...) at screenshothandler.d:536
#8  0x0000000000561be0 in asgen.handlers.screenshothandler.processScreenshot(asgen.result.GeneratorResult, appstream.Component.Component, appstream.Screenshot.Screenshot, immutable(char)[], uint) (gres=0x7f3a0bba5a00, cpt=0x7f3a0be42d80, scr=0x7f3a0bffdd80, mediaExportDir=..., scrNo=1) at screenshothandler.d:98
#9  0x0000000000561722 in asgen.handlers.screenshothandler.processScreenshots(asgen.result.GeneratorResult, appstream.Component.Component, immutable(char)[]) (gres=0x7f3a0bba5a00, cpt=0x7f3a0be42d80, mediaExportDir=...) at screenshothandler.d:53
#10 0x000000000052d777 in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7f4a3ed50100) at extractor.d:216
#11 0x00000000005188da in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
#12 0x000000000051db2c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
#13 0x00007f6a49729159 in std.parallelism.TaskPool.executeWorkLoop() () at /usr/lib/libphobos2-ldc.so.71
#14 0x00007f6a491f8a51 in thread_entryPoint () at /usr/lib/libdruntime-ldc.so.71
#15 0x00007f6a48752454 in start_thread () at /usr/lib/libpthread.so.0
#16 0x00007f6a48cc37df in clone () at /usr/lib/libc.so.6

Thread 6 (Thread 0x7f4a3d56b700 (LWP 1222)):
#0  0x00007f6a48cb64ed in read () at /usr/lib/libc.so.6
#1  0x00007f6a4aa382f0 in  () at /usr/lib/libarchive.so.13
#2  0x00007f6a4aa3236a in __archive_read_filter_ahead () at /usr/lib/libarchive.so.13
#3  0x00007f6a4aa3da41 in  () at /usr/lib/libarchive.so.13
#4  0x00007f6a4aa32758 in __archive_read_filter_consume () at /usr/lib/libarchive.so.13
#5  0x00007f6a4aa5a1f2 in  () at /usr/lib/libarchive.so.13
#6  0x00007f6a4aa31ced in archive_read_data_skip () at /usr/lib/libarchive.so.13
#7  0x00000000004d1285 in asgen.zarchive.ArchiveDecompressor.readData(immutable(char)[]) (this=..., fname=...)
    at zarchive.d:312
#8  0x00000000005a790d in asgen.backends.archlinux.alpkg.ArchPackage.getFileData(immutable(char)[]) (this=0x7f4a34c47900, fname=...) at alpkg.d:81
#9  0x000000000052c8c7 in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7f4a34c47900) at extractor.d:90
#10 0x00000000005188da in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
#11 0x000000000051db2c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope---Type <return> to continue, or q <return> to quit---
 int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
#12 0x00007f6a49729159 in std.parallelism.TaskPool.executeWorkLoop() () at /usr/lib/libphobos2-ldc.so.71
#13 0x00007f6a491f8a51 in thread_entryPoint () at /usr/lib/libdruntime-ldc.so.71
#14 0x00007f6a48752454 in start_thread () at /usr/lib/libpthread.so.0
#15 0x00007f6a48cc37df in clone () at /usr/lib/libc.so.6

Thread 5 (Thread 0x7f4a3cd6a700 (LWP 1223)):
#0  0x00007f6a48cb64ed in read () at /usr/lib/libc.so.6
#1  0x00007f6a48c4dad4 in __GI__IO_file_xsgetn () at /usr/lib/libc.so.6
#2  0x00007f6a48c42b29 in fread () at /usr/lib/libc.so.6
#3  0x00000000004d3282 in std.stdio.File.rawRead!(ubyte).rawRead(ubyte[]) (this=..., buffer=...)
    at /usr/include/dlang/ldc/std/stdio.d:929
#4  0x00007f6a49743842 in std.stdio.File.ByChunk.this(std.stdio.File, ubyte[]) () at /usr/lib/libphobos2-ldc.so.71
#5  0x00007f6a49743c2d in std.stdio.File.byChunk(ulong) () at /usr/lib/libphobos2-ldc.so.71
#6  0x00007f6a4972ff59 in std.process.executeImpl!(std.process.pipeProcess, const(char[])[]).executeImpl(const(char[])[], const(immutable(char)[][immutable(char)[]]), std.process.Config, ulong, const(char[])) () at /usr/lib/libphobos2-ldc.so.71
#7  0x00007f6a4972fe31 in std.process.execute(const(char[][]), const(immutable(char)[][immutable(char)[]]), std.process.Config, ulong, const(char[])) () at /usr/lib/libphobos2-ldc.so.71
#8  0x000000000056f57b in asgen.image.optimizePNG(immutable(char)[]) (fname=...) at image.d:64
#9  0x0000000000571698 in asgen.image.Canvas.savePng(immutable(char)[]) (this=0x7f3a0c066000, fname=...) at image.d:427
#10 0x0000000000557d2f in asgen.handlers.iconhandler.IconHandler.storeIcon(appstream.Component.Component, asgen.result.GeneratorResult, immutable(char)[], asgen.backends.interfaces.Package, immutable(char)[], asgen.utils.ImageSize) (this=0x7f4a0ef4c480, cpt=0x7f3a0be82540, gres=0x7f3a0bbbc000, cptExportPath=..., sourcePkg=0x7f4a3fe26a00, iconPath=..., size=...)
    at iconhandler.d:483
#11 0x0000000000558c7e in asgen.handlers.iconhandler.IconHandler.process(asgen.result.GeneratorResult, appstream.Component.Component) (this=0x7f4a0ef4c480, gres=0x7f3a0bbbc000, cpt=0x7f3a0be82540) at iconhandler.d:548
#12 0x000000000052d6d8 in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7f4a3fe26a00) at extractor.d:209
#13 0x00000000005188da in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
#14 0x000000000051db2c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
#15 0x00007f6a49729159 in std.parallelism.TaskPool.executeWorkLoop() () at /usr/lib/libphobos2-ldc.so.71
#16 0x00007f6a491f8a51 in thread_entryPoint () at /usr/lib/libdruntime-ldc.so.71
#17 0x00007f6a48752454 in start_thread () at /usr/lib/libpthread.so.0
#18 0x00007f6a48cc37df in clone () at /usr/lib/libc.so.6

Thread 4 (Thread 0x7f4a377fe700 (LWP 1225)):
#0  0x00007f6a48cb64ed in read () at /usr/lib/libc.so.6
#1  0x00007f6a4aa382f0 in  () at /usr/lib/libarchive.so.13
---Type <return> to continue, or q <return> to quit---
#2  0x00007f6a4aa3236a in __archive_read_filter_ahead () at /usr/lib/libarchive.so.13
#3  0x00007f6a4aa3da41 in  () at /usr/lib/libarchive.so.13
#4  0x00007f6a4aa32758 in __archive_read_filter_consume () at /usr/lib/libarchive.so.13
#5  0x00007f6a4aa5a1f2 in  () at /usr/lib/libarchive.so.13
#6  0x00007f6a4aa31ced in archive_read_data_skip () at /usr/lib/libarchive.so.13
#7  0x00000000004d1285 in asgen.zarchive.ArchiveDecompressor.readData(immutable(char)[]) (this=..., fname=...)
    at zarchive.d:312
#8  0x00000000005a790d in asgen.backends.archlinux.alpkg.ArchPackage.getFileData(immutable(char)[]) (this=0x7f4a2ddc2d00, fname=...) at alpkg.d:81
#9  0x000000000052d00c in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7f4a2ddc2d00) at extractor.d:159
#10 0x00000000005188da in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
#11 0x000000000051db2c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
#12 0x00007f6a49729159 in std.parallelism.TaskPool.executeWorkLoop() () at /usr/lib/libphobos2-ldc.so.71
#13 0x00007f6a491f8a51 in thread_entryPoint () at /usr/lib/libdruntime-ldc.so.71
#14 0x00007f6a48752454 in start_thread () at /usr/lib/libpthread.so.0
#15 0x00007f6a48cc37df in clone () at /usr/lib/libc.so.6

Thread 3 (Thread 0x7f4a36ffd700 (LWP 1226)):
#0  0x00007f6a48cb64ed in read () at /usr/lib/libc.so.6
#1  0x00007f6a4aa382f0 in  () at /usr/lib/libarchive.so.13
#2  0x00007f6a4aa3236a in __archive_read_filter_ahead () at /usr/lib/libarchive.so.13
#3  0x00007f6a4aa3da41 in  () at /usr/lib/libarchive.so.13
#4  0x00007f6a4aa32758 in __archive_read_filter_consume () at /usr/lib/libarchive.so.13
#5  0x00007f6a4aa5a1f2 in  () at /usr/lib/libarchive.so.13
#6  0x00007f6a4aa31ced in archive_read_data_skip () at /usr/lib/libarchive.so.13
#7  0x00000000004d1285 in asgen.zarchive.ArchiveDecompressor.readData(immutable(char)[]) (this=..., fname=...)
    at zarchive.d:312
#8  0x00000000005a790d in asgen.backends.archlinux.alpkg.ArchPackage.getFileData(immutable(char)[]) (this=0x7f4a3476db00, fname=...) at alpkg.d:81
#9  0x000000000052d00c in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7f4a3476db00) at extractor.d:159
#10 0x00000000005188da in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
#11 0x000000000051db2c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
#12 0x00007f6a49729159 in std.parallelism.TaskPool.executeWorkLoop() () at /usr/lib/libphobos2-ldc.so.71
#13 0x00007f6a491f8a51 in thread_entryPoint () at /usr/lib/libdruntime-ldc.so.71
---Type <return> to continue, or q <return> to quit---
#14 0x00007f6a48752454 in start_thread () at /usr/lib/libpthread.so.0
#15 0x00007f6a48cc37df in clone () at /usr/lib/libc.so.6

Thread 2 (Thread 0x7f4a3dd6c700 (LWP 1221)):
#0  0x00007f6a48c92ffd in nanosleep () at /usr/lib/libc.so.6
#1  0x00007f6a491f97ed in core.thread.Thread.sleep(core.time.Duration) () at /usr/lib/libdruntime-ldc.so.71
#2  0x00007f6a49201c82 in core.internal.spinlock.SpinLock.lock() shared () at /usr/lib/libdruntime-ldc.so.71
#3  0x00007f6a49212cbe in gc.gc.GC.runLocked!(gc.gc.GC.extendNoSync(void*, ulong, ulong, const(TypeInfo)), gc.gc.extendTime, gc.gc.numExtends, void*, ulong, ulong, const(TypeInfo)).runLocked(ref void*, ref ulong, ref ulong, ref const(TypeInfo)) ()
    at /usr/lib/libdruntime-ldc.so.71
#4  0x00007f6a49212c71 in gc.gc.GC.extend(void*, ulong, ulong, const(TypeInfo)) () at /usr/lib/libdruntime-ldc.so.71
#5  0x00000000004ded4d in std.array.Appender!(char[]).Appender.ensureAddable(ulong) (this=..., nelems=1)
    at /usr/include/dlang/ldc/std/array.d:2773
#6  0x00000000004de71e in std.array.Appender!(char[]).Appender.put!(char).put(char) (this=..., item=47 '/')
    at /usr/include/dlang/ldc/std/array.d:2833
#7  0x00000000004dbfeb in std.array.array!(std.path.asNormalizedPath!(const(char)[]).asNormalizedPath(const(char)[]).Result).array(std.path.asNormalizedPath!(const(char)[]).asNormalizedPath(const(char)[]).Result) (r=...)
    at /usr/include/dlang/ldc/std/array.d:135
#8  0x00000000004d08f0 in std.path.buildNormalizedPath!(char).buildNormalizedPath(const(char[])[]...) (paths=...)
    at /usr/include/dlang/ldc/std/path.d:1597
#9  0x00000000004d06ce in asgen.zarchive.ArchiveDecompressor.pathMatches(immutable(char)[], immutable(char)[]) (this=..., path1=..., path2=...) at zarchive.d:198
#10 0x00000000004d0e42 in asgen.zarchive.ArchiveDecompressor.readData(immutable(char)[]) (this=..., fname=...)
    at zarchive.d:283
#11 0x00000000005a790d in asgen.backends.archlinux.alpkg.ArchPackage.getFileData(immutable(char)[]) (this=0x7f4a34921c00, fname=...) at alpkg.d:81
#12 0x000000000052c8c7 in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7f4a34921c00) at extractor.d:90
#13 0x00000000005188da in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
#14 0x000000000051db2c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
#15 0x00007f6a49729159 in std.parallelism.TaskPool.executeWorkLoop() () at /usr/lib/libphobos2-ldc.so.71
#16 0x00007f6a491f8a51 in thread_entryPoint () at /usr/lib/libdruntime-ldc.so.71
#17 0x00007f6a48752454 in start_thread () at /usr/lib/libpthread.so.0
#18 0x00007f6a48cc37df in clone () at /usr/lib/libc.so.6

Thread 1 (Thread 0x7f4a3e56d700 (LWP 1220)):
#0  0x0000000000590fff in asgen.result.GeneratorResult.finalize() (this=0x7f3a0bbbc100) at result.d:252
#1  0x000000000052d839 in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7f---Type <return> to continue, or q <return> to quit---
4a3fe95400) at extractor.d:234
#2  0x00000000005188da in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
#3  0x000000000051db2c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
#4  0x00007f6a49729159 in std.parallelism.TaskPool.executeWorkLoop() () at /usr/lib/libphobos2-ldc.so.71
#5  0x00007f6a491f8a51 in thread_entryPoint () at /usr/lib/libdruntime-ldc.so.71
#6  0x00007f6a48752454 in start_thread () at /usr/lib/libpthread.so.0
#7  0x00007f6a48cc37df in clone () at /usr/lib/libc.so.6

[antonio-rojas]

Last lines of verbose output before the segfault:

2017-02-23 12:03:09 - INFO: Scanned plasma-desktop/5.9.2-1/x86_64, could be interesting.
2017-02-23 12:03:09 - INFO: Scanned kalzium/16.12.2-1/x86_64, could be interesting.
2017-02-23 12:03:09 - INFO: Scanned kile/2.1.3-7/x86_64, could be interesting.
2017-02-23 12:03:09 - INFO: Scanned appstream/0.10.6-1/x86_64, could be interesting.
2017-02-23 12:03:09 - INFO: Scanned evolution-ews/3.22.5-1/x86_64, could be interesting.
2017-02-23 12:03:09 - INFO: Scanned kdebugsettings/16.12.2-1/x86_64, could be interesting.
2017-02-23 12:03:09 - DEBUG: Creating new IconHandler
2017-02-23 12:03:09 - DEBUG: Opening contents cache.
2017-02-23 12:03:09 - DEBUG: Created new IconHandler.

[ximion]

This crash makes no sense, since the cpts array is GC-managed and accessible (the GeneratorResult instance is alive).
Could be some weird threading bug, but I doubt it. The log output unfortunately doesn't tell us which package was processed, since the data processing happens in parallel and only after it is done a log message will be written.

I was seeing some strange crashes as well (broken string in libappstream which just can't be broken) yesterday, but I was attributing those to the latest Git master patch on scoped classes so far. Maybe something in asgen is messing with us...

[antonio-rojas]

I've added some debug. Seems that the culprit is archlinux-menus
https://www.archlinux.org/packages/extra/any/archlinux-menus/

[antonio-rojas]

Replacing line 252 in result.d with

foreach (cpt; cpts.dup.byValue ()) {

fixes the problem

[ximion]

So, something destroys the cpts array... Weird.

[ximion]

Can you maybe give me the full backtrace (bt full) on the position where it crashed? Duplicating the array is only masking the issue.
Also, if you change "cpts.byValue" with "cpts.values", does that make the crash disappear?

[antonio-rojas]

Yes, cpts.values also fixes the crash.

Full backtrace: (there's some line number offset due to debug lines I added)

(gdb) bt full
#0  0x0000000000592405 in asgen.result.GeneratorResult.finalize() (this=0x7f808ac57b80) at result.d:255
        customHashTable = 0x7f808fa40100
        noCustomKeysAllowed = true
        __r3563 = {r = {impl = 0x7f808abdbd00 "\b", idx = 21}}
        conf = 0x7fb0e8898400
        cpt = <optimized out>
        desc_added = <optimized out>
#1  0x000000000052eb19 in asgen.extractor.DataExtractor.processPackage(asgen.backends.interfaces.Package) (this=..., pkg=0x7fb0e8930500)
    at extractor.d:235
        gres = 0x7f808ac57b80
        desktopFiles = 0x7f808abca980 " "
        metadataFiles = 0x0
        mdata = 0x7f808abcaa80
        hasFontComponent = false
#2  0x0000000000519cba in asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package) (pkg=<optimized out>) at engine.d:139
        pkid = "archlinux-menus/2.5-3/any"
        res = 0x7f808cdbba00
        __sync3734 = 0x7fb0e8899900
#3  0x000000000051ef0c in std.parallelism.ParallelForeach!(asgen.backends.interfaces.Package[]).ParallelForeach.opApply(scope int(ref asgen.backends.interfaces.Package) delegate).doIt() () at /usr/include/dlang/ldc/std/parallelism.d-mixin-3785:3829
        dg = 
          {context = 0x7ffd944d8a88 "", funcptr = 0x519c40 <asgen.engine.Engine.processPackages(asgen.backends.interfaces.Package[], asgen.handlers.iconhandler.IconHandler).__foreachbody3(ref asgen.backends.interfaces.Package)>}
        i = 2439
        __key3775 = 2439
        __limit3776 = 2522
        end = 2522
        this = <optimized out>
        myUnitIndex = 25
        start = 2425
#4  0x00007fb0e64f4159 in std.parallelism.TaskPool.executeWorkLoop() () at /usr/lib/libphobos2-ldc.so.71
#5  0x00007fb0e5fc3a51 in thread_entryPoint () at /usr/lib/libdruntime-ldc.so.71
#6  0x00007fb0e551d454 in start_thread () at /usr/lib/libpthread.so.0
#7  0x00007fb0e5a8e7df in clone () at /usr/lib/libc.so.6

[ximion]

This smells a bit like a D (DRuntime / LDC) bug, I can't make out any issue in the code. byValue should return a range on the cpts associative array which is still alive at that time (it must be, otherwise we couldn't even dup it), yet, we get a segfault when calling it.
It's also pretty weird that this happens with one specific package... Have you tried this a few times and is it always the same package?
Can you maybe also try to wrap the finalize call in a synchronized block, to rule out some other thread messing with the data? (shouldn't happen at all, there is no such code anywere - but just to be sure).
Just change https://github.com/ximion/appstream-generator/blob/master/src/asgen/extractor.d#L235 into

synchronized gres.finalize ();

I unfortunately still can't reproduce this issue, but squashed another, different, bug in AppStream in the process of poking the code a little and running it on a Debian repo :P

[ximion]

Oh damn... While writing a blogpost to the D language forums about this weird behavior, I found the issue: Recently, a new hint type was added, which I thought does not trigger the rejection of a package - turns out it does.
This means that the array is actually manipulated in this loop, leading to the crash.

I have no idea why I haven't seen this before, obvious issue is obvious (there is even a comment in the code warning about this...).
Thanks for reporting the issue, will be fixed soon (in a few minutes :P)

[ximion]

Maybe have asgen reprocess the data after updating to the new release, I wouldn't necessarily trust it now (some bad data might have slipped through).
Simply run asgen remove-found <suite> prior to processing the archive, that will save you from excessive reprocessing of every single package.