ActiveStorage Mirror Issue

[airjoshb]

I am trying to set up ActiveStorage mirror so that I can migrate from S3 into my Cloudinary. When I try to run the mirror check and upload, I'm getting an error, "RuntimeError (Must supply cloud_name)"

If I check Cloudinary.config, I get all the correct info:
#<OpenStruct cloud_name="airjoshb", api_key="my_key", api_secret="my_key", private_cdn=false, false=false, secure=true, enhance_image_tag=true, static_file_support=false>

But if I check Cloudinary.config.cloud_name, it returns false

If I try to run the mirror command locally, I get a bit more info in that it is confirming the file isn't in my mirror on Cloudinary
Cloudinary Storage (4.0ms) Checked if file exists at key: 3uxbbrz9kh4mw7iq2z2xzyuo1w2z (no) Mirror Storage (4.5ms) Mirrored file at key: 3uxbbrz9kh4mw7iq2z2xzyuo1w2z (checksum: DSAVwrDMF+tj9ncc9pCX2A==) /Users/joshua/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/cloudinary-2.2.0/lib/cloudinary/api.rb:1269:in call_api': Must supply cloud_name (RuntimeError)`

Any thoughts on why it finds the cloud_name but is returning false when I try to use it?

ruby 3.0.0p0
Rails 7.0.4
cloudinary (2.2.0)
activestorage (7.0.4)

I have tried all manner of configurations, using the cloudinary.yml in the config folder

development:
    cloud_name: airjoshb
    api_key: ENV.fetch('CLOUDINARY_KEY_ID')
    api_secret: ENV.fetch('CLOUDINARY_ACCESS_KEY')
    enhance_image_tag: true
    static_file_support: false

  production:
    cloud_name: airjoshb
    api_key: ENV.fetch('CLOUDINARY_KEY_ID')
    api_secret: ENV.fetch('CLOUDINARY_ACCESS_KEY')
    enhance_image_tag: true
    static_file_support: true

In an initializer, cloudinary.rb where the env includes the key:key@cloud_name format

require 'cloudinary'

Cloudinary.config_from_url("cloudinary://ENV.fetch('CLOUDINARY_URL')")
Cloudinary.config do |config|
  config.secure = true
  config.enhance_image_tag = true
  config.static_file_support= false
end

And, my storage.yml is super simple, as the gem seems to prefer using the initializer

cloudinary:
  service: Cloudinary
  folder: switch-bakery

# Use bin/rails credentials:edit to set the AWS secrets (as aws:access_key_id|secret_access_key)
amazon:
  service: S3
  access_key_id:     <%= ENV.fetch('AWS_ACCESS_KEY_ID') %>
  secret_access_key: <%= ENV.fetch('AWS_SECRET_ACCESS_KEY') %>
  region:            'us-west-1'
  bucket: <%= ENV.fetch('S3_BUCKET_NAME') %>

production:
  service: Mirror
  primary: amazon
  mirrors:
    - cloudinary

[const-cloudinary]

@airjoshb , thank you for reporting the issue!

This is indeed weird.

Did it happen in the previous version of the SDK, for example: 2.1.2 ?

In the latest version we added explicit ostruct dependency, maybe that could affect something.

In addition you can also try to pass cloud_name, api_key and api_secret in your storage.yml

cloudinary:
  service: Cloudinary
  folder: switch-bakery
  cloud_name: airjoshb
  api_key: ENV.fetch('CLOUDINARY_KEY_ID')
  api_secret: ENV.fetch('CLOUDINARY_ACCESS_KEY')

[airjoshb]

Thanks for getting back to me. I hadn't tried this particular setup in the previous SDK. I'll try downgrading and see if that works. I have another project that could be on the older SDK and that works flawlessly, which is why I have been so stumped! I did try setting everything in the storage.yml with the same effect.

One difference I noticed between the two in calling the config is that in the working project, there is no false=false, which shows up no matter how I configure in the newer setup ... maybe a hint?

#<OpenStruct cloud_name="airjoshb", api_key="my_key", api_secret="my_key", private_cdn=false, false=false, secure=true, enhance_image_tag=true, static_file_support=false>

[const-cloudinary]

@airjoshb when setting those values in storage.yml those values are not passed to the global config, but stored as @options in the service class and then they get passed to the SDK directly.

[const-cloudinary]

@airjoshb I was able to setup the mirror storage, set a few different cloudinary configurations in storage.yml

mirror:
  service: Mirror
  primary: local
  mirrors:
    - cloudinary_test
    - cloudinary_development

And both worked for me.

[airjoshb]

Ok- I was able to get the direct mirroring to work using the storage.yml configuration. The point of setting this up was to migrate from S3 to Cloudinary and so I was using the mirror_later feature of active storage to accomplish this, but I get the same error for cloud_name when I run it. Current status:

1. When I upload a new image attached to my post using activestorage, it appears that I get the image in the primary (S3) and mirror (Cloudinary);
2. However, when I try to run mirror_later (eg. ActiveStorage::Blob.last.mirror_later), I get the following output saying that authentication fails because there is no cloud_name

2024-10-04T01:08:06.211Z pid=1383246 tid=s7bi class=ActiveStorage::MirrorJob jid=0cb5a401181cc652f98deeb7 elapsed=0.108 INFO: fail
2024-10-04T01:08:06.211Z pid=1383246 tid=s7bi WARN: {"context":"Job raised exception","job"
{"retry":true,"queue":"default","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActiveStorage::MirrorJob","args":[{"job_class":"ActiveStorage::MirrorJob","job_id":"064cd43c-d884-47fe-be51-f459ac92211e","provider_job_id":null,"queue_name":"default","priority":null,"arguments":["pxtkb9kg0qmzzkihjnacb5uv9orb",{"checksum":"Eev/0M66Yt6UobSCESH3sA==","_aj_ruby2_keywords":
["checksum"]}],"executions":0,"exception_executions":{},"locale":"en","timezone":"Pacific Time (US & Canada)","enqueued_at":"2024-10-04T00:59:40Z"}],"jid":"0cb5a401181cc652f98deeb7","created_at":1728003580.2792048,"enqueued_at":1728004086.1002667,"error_message":"unknown 
api_key","error_class":"Cloudinary::BaseApi::AuthorizationRequired","failed_at":1728003581.616552,"retry_count":4,"retried_at":1728003794.3334181}}
2024-10-04T01:08:06.211Z pid=1383246 tid=s7bi WARN: RuntimeError: Must supply cloud_name

And to ensure clarity, I am passing the cloud_name in the storage.yml, per your original comment. Thanks for all the help!

[airjoshb]

hi @const-cloudinary - any additional thoughts on this?

[const-cloudinary]

Hey @airjoshb ,

There are actually 2 ways configuring Cloudinary using env variables, using CLOUDINARY_URL and using separate env variables, see:

cloudinary_gem/lib/cloudinary/config.rb

Line 9 in c50fa14

if ENV["CLOUDINARY_CLOUD_NAME"]

I would check CLOUDINARY_CLOUD_NAME env variable in your environment.

[airjoshb]

Thanks, @const-cloudinary

This all feels quite strange, as I am now getting an error: Cloudinary::BaseApi::AuthorizationRequired: unknown api_key

I have taken your advice and set up env variables two different ways to see if it would make a difference, one with all of the ENV variables separate

irb(main):004:0> ENV.keys.select! { |key| key.start_with? "CLOUDINARY_" }
=> ["CLOUDINARY_API_KEY", "CLOUDINARY_API_SECRET", "CLOUDINARY_CLOUD_NAME"]

and get the proper keys, matching up to the API keys in the console

Cloudinary.config
=> #<OpenStruct api_key="xxxxxx", api_secret="xxxxxx", cloud_name="airjoshb">

Running mirror_later, the job is now enqueued successfully

INFO -- : [ActiveJob] Enqueued ActiveStorage::MirrorJob (Job ID: efe8eaa4-ab0f-41ad-a032-4a631d4d09ae) to Sidekiq(default) with arguments: "kr7r5oq5wv7l0rsh5asbn29q3xwb", {:checksum=>"J0U1SzGAX1sv3mYTXmb46g=="}
=> #<ActiveStorage::MirrorJob:0x00005640c20849f0 @arguments=["kr7r5oq5wv7l0rsh5asbn29q3xwb", {:checksum=>"J0U1SzGAX1sv3mYTXmb46g=="}], @job_id="efe8eaa4-ab0f-41ad-a032-4a631d4d09ae", @queue_name="default", @priority=nil, @executions=0, @exception_executions={}, @timezone="Pacific Time (US & Canada)", @successfully_enqueued=true, @provider_job_id="6abf7428301c1027f05de98a">

but when the job runs I get, Cloudinary::BaseApi::AuthorizationRequired: unknown api_key

I get the same error if setting up using CLOUDINARY_URL

irb(main):001:0> ENV["CLOUDINARY_CLOUD_NAME"]
=> nil
irb(main):005:0> ENV["CLOUDINARY_URL"]
=> "cloudinary://xxxxx:xxxx@airjoshb"
irb(main):006:0>Cloudinary.config
=> #<OpenStruct cloud_name="airjoshb", api_key="xxxxx", api_secret="xxxxx", private_cdn=false>

I was thinking that perhaps the key wasn't working, but the uploading works directly in the mirror. Thanks for your patience and help!

[const-cloudinary]

@airjoshb, this is really strange.

One thing I can think of is that maybe ActiveStorage::MirrorJob is running in a separate process that does not inherit environment variables and that's why nothing is initialized.

Maybe try overriding the default MirrorJob as follows:

class ActiveStorage::MirrorJob < ApplicationJob
  queue_as :default

  def perform(blob, attachable)
    # Custom behavior for mirroring
    Rails.logger.info "Custom MirrorJob: Starting mirroring for #{blob.key}"

    Cloudinary.config do |config|
        config.cloud_name = ENV['CLOUDINARY_CLOUD_NAME']
        config.api_key = ENV['CLOUDINARY_API_KEY']
        config.api_secret = ENV['CLOUDINARY_API_SECRET']
     end

    # Call the original mirroring process
    super if defined?(super)
  end
end

Or just print variables and check what you see there.

Another way would be to override CloudinaryService

module ActiveStorage
  class Service::CustomCloudinaryService < Service::CloudinaryService
    def initialize(**options)
      # Explicitly set @options to include credentials
      @options = options.merge({
        cloud_name: ENV['CLOUDINARY_CLOUD_NAME'],
        api_key: ENV['CLOUDINARY_API_KEY'],
        api_secret: ENV['CLOUDINARY_API_SECRET'],
        secure: true # or any other Cloudinary settings you want to configure
      })

      # Call the original initializer to make sure CloudinaryService is set up properly
      super(**@options)
    end
  end
end

and then in your storage.yml

cloudinary_custom:
  service: CustomCloudinaryService

Similarly you can modify

def upload(key, io, checksum: nil, **options)

or other methods and pass credentials explicitly.

Please let me know if it works for you.