Having problems uploading an ISO to CS

[rongage]

Oddly enough, this is not the usual complaints about not being able to upload an ISO.

The problem appears to be that there is an exception is being thrown on the SSVM after the upload actually finishes. The follow on problem then is that the uploaded ISO never goes to "ready".

The text of the exception on the SSVM:
2024-10-12 00:50:45,408 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-4:null) BODY FileUpload: FileUpload: content-disposition: form-data; name="files[]"; filename="win11pro.iso"
content-type: application/octet-stream; charset=UTF-8
content-length: 6852902912
Completed: true
IsInMemory: false
RealFile: /mnt/SecStorage/7af1bb47-92e6-3081-975a-784b7e64c293/template/tmpl/2/209/FUp_2984701520707360110_-838595071 DeleteAfter: false
2024-10-12 00:50:45,688 INFO [storage.template.OVAProcessor] (nioEventLoopGroup-3-4:null) Template processing. templatePath: /mnt/SecStorage/7af1bb47-92e6-3081-975a-784b7e64c293/template/tmpl/2/209, templateName: 5665cb79-d178-4125-8820-553303560ccf
2024-10-12 00:50:45,692 INFO [storage.template.OVAProcessor] (nioEventLoopGroup-3-4:null) Unable to find the vmware template file: /mnt/SecStorage/7af1bb47-92e6-3081-975a-784b7e64c293/template/tmpl/2/209/5665cb79-d178-4125-8820-553303560ccf.ova
2024-10-12 00:50:45,692 INFO [storage.template.VmdkProcessor] (nioEventLoopGroup-3-4:null) Template processing. templatePath: /mnt/SecStorage/7af1bb47-92e6-3081-975a-784b7e64c293/template/tmpl/2/209, templateName: 5665cb79-d178-4125-8820-553303560ccf
2024-10-12 00:50:45,695 INFO [storage.template.VmdkProcessor] (nioEventLoopGroup-3-4:null) Unable to find the vmware template file: /mnt/SecStorage/7af1bb47-92e6-3081-975a-784b7e64c293/template/tmpl/2/209/5665cb79-d178-4125-8820-553303560ccf.vmdk
2024-10-12 00:50:45,734 WARN [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-4:null) . Exception occurred: refCnt: 0, decrement: 1
2024-10-12 00:50:45,739 WARN [netty.channel.AbstractChannelHandlerContext] (nioEventLoopGroup-3-4:null) An exception 'java.lang.NullPointerException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:148)
at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:316)
at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:993)
at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:969)
at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:243)
at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-10-12 00:51:25,486 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Determined host 192.168.0.171 corresponds to IP 192.168.0.171
2024-10-12 00:52:25,600 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) Determined host 192.168.0.171 corresponds to IP 192.168.0.171

I'll try restarting the SSVM, then if that doesn't work, I'll try regenerating the SSVM.

[rongage]

Sure enough, neither resetting the SSVM or regenerating it helped at all. Still getting the exception at the end of the upload and the uploaded file never switches to "ready".

[rongage]

While I don't know for certain if this was the problem, I did find a duplicate entry in database table vm_template. I was able to delete all the entries after the duplicate entry, including FK references in template_spool_ref and template_store_ref. I'm trying another ISO upload now. Results soon.

[rongage]

Nope - ISO uploads are still broken.

[weizhouapache]

@rongage
which ACS version do you use ? can you try latest 4.19.1.X ?