Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feedback from CERN, followup ocConf 2019 #43

Closed
labkode opened this issue Oct 30, 2019 · 8 comments
Closed

Feedback from CERN, followup ocConf 2019 #43

labkode opened this issue Oct 30, 2019 · 8 comments
Labels
Early-Adopter:CERN Type:Discussion

Comments

@labkode
Copy link
Contributor

labkode commented Oct 30, 2019

Overview

How shares are organized and synchronized (folder can be moved around etc). Looking at "shared with me" list now: it's too long and cumbersome to manage for some users. This needs a proper product design and scalability discussion, being able to filter out by different fields, hidding and unhidding the shares, filtering by user, group or ocm types.
Also we'd like that from the web UI we can decide if this share is synced to the mobile or to the desktop client automatically as we have the information about what devices the user is using.

For phoenix, we'd like that becomes a platform which we can cleanly extend with our own views or plugins (that may later be merged with upstream if of general interest). It is very important to keep consistent the product aspects (such as sharing) across different devices interfaces (phoenix & mobile for example)? There will always be differences due to specifics of each device but some concepts should be the same (and documented somewhere, it does not belong to the API really).

Related to CS3 APIs, for Q4 2019, CERN will provide a prototype of the grpc connection for Phoenix and ownCloud one for the desktop sync client.

Accepting shares

  • cern: accepting shares is possible from the web interface, missing in mobile and desktop clients
  • oc: plans to have it done also for mobile and desktop. The new IOS app already allows that.

File Metadata propagation for the desktop sync client

  • cern: we'd like to sync execution bit and Mac Os labels for example. To be enabled by exposing a new capability.
  • oc: understand first how the CS3 APIs will fit this purpose. cern to create an issue to follow up.

Syncing unreadable folders

  • cern: oc expects a tree with read-write to be completely read-write, but sometimes there will be folders with more restrictive permissions, like read, therefore the sync client will abort the sync for the entire folder. TODO(labkode): find isue number. TODO(cern): test behaviour with 2.6.0-rc2

Disable user sharing for individual files

  • cern: we'd like to configure clients (mobile, desktop, web) to avoid creating shares to users on individual files.
  • oc: plan to do it by offering a new API that will be consumed by clients to render the correct dialogs for sharing.

Context menu for applications in desktop sync client

  • cern: the desktop sync client offers an OS-integrated context menu to share. We'd like to add our own entries to enable "open in Office" for example.
  • oc: won't add support for context menus. We'are going in the direction to open directly on the oc UI, for example, by clicking in the desktop show me versions, it will open a dialog with a browser inside rendering the versions tabs found in the oc UI. Idea is to extend this behaviour to applications, so "Open in Office" will open the file also in the browser.
    OCIS could provide views specially targeted to the desktop client making the integratin with the OS more native.

Old chunking algorithm

  • cern: still relying on old chunking.
  • oc: still in 2.7 but no support if something breaks.

Request ID header

  • oc: sync client currently sends a request ID in the X-Request-Id header
  • cern: it can also be interesting if client could send extra information by using OpenCensus instrumentation (like start of the request seen from sync client, version number, ...) that will give us lot of insights from a client perspective rather than server-side.

UI Private link

  • cern: we'd like to allow people to just copy/paste the URL to access a shared resource.
  • oc: plan to merge private link and public link into one concept. Maier as product manager liked the idea of people just copy/pasting urls for sharing.

Configuration management for sync client: major priority for cern

  • cern: we want to dynamically provision users with configuration (hidden files, enabled features, ...). Currently we can configure at build time some constraints but that is not enough, we'd like to have it dynamically per user and easily changeable by and administrator without re-building a branded client and distributing it.
  • oc: current possibility is statically-branded builds with ownbrander and MDM-style deployment for mobile. There are some plans for user profiles but they will come with OCIS and new APIs that will allow us to achieve such functionality. Michael with follow up with configuration team as he wants to converge all configuration options into a single namespace to brand all clients the same way. Once that is there, the dynamic configuration will follow.

Recall of an user desktop sync client configuration

  • cern: we'd like to recall a user configuration as an administrator to not ask the user to find hidden special files (.sqlite, .ocsync, ...) to send into a ticket.
  • oc: we currently have a way where a user can be asked to create a debug archive that currently contains only the log information. cern will send list of files to be included in the archive. The generated archive (zip) is generated and the user chooses where to store it.
  • cern: it would be very helpful if the archive is stored automatically in the sync folder so the archive arrives to the administrator automatically.
  • oc: Hannha mentioned the possibily to put a magic file into the synced folder to trigger that behaviour. To be follow up in a subsequent meeting. The feature of pressing F12 to get the logs has been disabled, the user has now the power to generate such archive at prefered location.

Symlinks synchronization

  • cern: we'd like to have the option to also synchronize a symbolic link
  • oc: it will be possible with the integration of ocis and the cs3apis

Shared journal for desktop sync client

  • cern: the current sync client stores one sync journal (logs, sqlitedb) per sync folder pair, having one is desired for debugging done by administrators.
  • oc: the new archive can get all those files.

Add a new sync folder pair desktop sync client wizard

  • cern: the current wizard is confusing for users as EOS evaluates permissions at a folder level not in a hirarchy, this causes some propfind requests to fail.
  • oc: cern to send detailed issue.

Update of Windows client

  • cern: update to 2.5* requires moving first to 2.4.3
  • oc: make sure cern has autoupdater on latest version

Virtual filesystem: major priority for cern

  • cern: we'd like that this feature reaches a production state.
  • oc: we have now a new version integrated natively with the Windows operative system like Dropbox and Google Drive does using native "cloud" APIs. We plan to the same for Mac OS Catalina. For Linux OS there is no clear path yet but we also want to offer this integration. Suggest to try with 2.6.0-rc2 and give feedback.
  • cern : we'll start testing it.
  • cern: tests so far didn't manange to connect to the production cluster by current capabilities.

Sync client syncs only UTF-8

  • cern: sync client converts latin and other encodings back to utf-8
  • oc: the user will could be notified that the file has been re-encoded to utf-8

Mobile client developments

  • oc: working on a new IOS application with better integration thans to IOS "FileProvider". Android application
    is getting re-implemented with a new architecture.
  • cern: will send issues found by users to see if they have been addressed. A requested feature is to have automatic synced folders/files, like a desktop sync client in the mobile.
  • oc: we plan to provide this functionality by having the same sync algorithm on the client.

Collaboration on Phoenix

  • oc: the UX needs to be improved. Efforts were done on functionality first. Focusing on improving the interface now.

  • cern: we have the concept of project spaces, that are collaborative spaces where people can work together.

  • oc: we also want to introduce such concept as "space rooms"

  • cern: that is a great idea, however the APIs for Phoenix need to be flexible enough to allow adding arbitrary new storage spaces without having to hack code. For example we exposed at some point in the past an application called "Global EOS" that allowed access to arbitrary storage pools.

  • cern: we want to allow users to copy/paste the url to access shared spaces. The motivation behind is that all our end-user targeted storage systems (DFS, AFS, EOS) expose a global path that allows people to copy/paste a path and gets access to the resource. The Web UI masks this path to the root url (/) having no way to allow such functionality (the workaround was the private link). We'd like that Phoenix brings this possibility either as a configuration option or as main product feature. This changes the semantic (by path) and also exposes parent directory names. This may or may not be desired, depending on context. So a holistic product design needed here as well.

  • oc: Patrick as product manager liked that concept. To be followed up on a subsequent call.

  • cern: the owncloud SDK and Phoenix should bring the possibility to configure the SDK with a different storage basename, currently all operations are mapped to "/", but we might want to send to "/eos/user/g/gonzalhu", that way the breadcrumbs for the user will be shown in the UI, that is a pre-requisite to allow copy/paste urls.

  • oc: Thomas and Vince understood the requirement and it should be possible to do so in the ownCloud SDK, they will follow up on a github issue. CERN will provide feedback by implementing the CS3APIs connector to the ownCloud SDK.

  • oc: some metadata attributes are prefixed with oc namespaces, like "https://owncloud/ns/fileid", how they will fit in the CS3APIs calls?

  • cern: the same prefix will be used in the CS3APIs when they are vendor-specific, so there should be no problem. For more general attributes we'll use some exisitng date dictionaries. cern to check with digital repositories at CERN for attribute classification.

Collaboration on OCIS/REVA

  • oc: released a first version of the OCIS setup where services are split in various repositories: ocis, ocis-webdav, ocis-phoenix, ocis-reva. Currenlty not using any micro-services framework. Will like to move to go-kit framewok as micro is becoming more comercial.

  • cern: agreed that the choice of framework for OCIS should be based on the sustenaibiliy of the framework for long term and avoiding projects led by single individuals.

  • cern: the current ocis setup is enough to run Reva alongside oc components as the connection between ocis-reva services and ocis-webdav is done by using the CS3 APIS. Current setup allows for evolution of Reva and OCIS independently while ensuring compatibily of using Ocis with any version of Reva. However, it could also be interesting to see if Reva could be backwards compatible with internal OCIs services (pkg commands).

  • oc: that could be possible if Reva uses the same framework and components as ocis. Thomas could prototype it.

  • cern: like the idea, however needs further discussion with cernbox team and follow up in a subsequent call. Worry that multiple repository approach will lead to code duplication and API checks are spread across services.

  • oc: code duplication will be there but relying on code generation should reduce the burden and for the API check a common library could be created.

  • cern: would like to follow the approach done with Restic for documentation: code lives alongside documentation and changelog of the software also. Also we'd like to tag a version and to have an automatic release.

  • oc: should be also doable in Drone and using the calens tool.

  • cern: done, thanks to Thomas for instructions.

CS3 APIS

  • cern: would like the cs3 apis to be compiled and publish to various languages.

  • oc: should be possible to achieve that by using Drone as the CI/CD service.

  • cern: done, thanks to Thomas for providing us intructions on how to do the setup.

  • oc: would be nice to have a proposal system for the CS3 APIs. In Javascript world there is TC39 for such discussions.

  • cern: would be make a lot of sense once the CS3APIs gets to a stable state and start to be used elsewhere. The current contributor guidelines should suffice for until then

  • oc: not many documentation of the CS3APIS

  • cern: working on a white paper on the CS3APIs to present the on CHEP, further documentation will follow alongside Reva.
@labkode labkode changed the title Feedback from CERN Feedback from CERN, followup ocConf 2019 Oct 30, 2019
@butonic butonic self-assigned this Nov 11, 2019
@butonic
Copy link
Member

butonic commented Nov 15, 2019 •
edited by micbar
Loading

I pinged the different people to get some feedback below and link to appropriate tickets. After that is done I will close the issue. Followups should happen in the corresponding issues. @labkode I recommend you should subscribe to the linked issues to track their progress.

Overview

How shares are organized and synchronized (folder can be moved around etc). Looking at "shared with me" list now: it's too long and cumbersome to manage for some users. This needs a proper product design and scalability discussion, being able to filter out by different fields, hidding and unhidding the shares, filtering by user, group or ocm types.

  • @pmaier1 needs issue, I know we are working on sharing ui concepts

Also we'd like that from the web UI we can decide if this share is synced to the mobile or to the desktop client automatically as we have the information about what devices the user is using.

  • involves registering devices, which is related to oidc, because each devices will have its own token
  • make the ocis oidc implementation work with the client test client VFS with ocis #52

For phoenix, we'd like that becomes a platform which we can cleanly extend with our own views or plugins (that may later be merged with upstream if of general interest). It is very important to keep consistent the product aspects (such as sharing) across different devices interfaces (phoenix & mobile for example)? There will always be differences due to specifics of each device but some concepts should be the same (and documented somewhere, it does not belong to the API really).

Related to CS3 APIs, for Q4 2019, CERN will provide a prototype of the grpc connection for Phoenix and ownCloud one for the desktop sync client.

  • @labkode this needs an issue somewhere

Accepting shares

  • cern: accepting shares is possible from the web interface, missing in mobile and desktop clients
  • oc: plans to have it done also for mobile and desktop. The new IOS app already allows that.

File Metadata propagation for the desktop sync client

  • cern: we'd like to sync execution bit and Mac Os labels for example. To be enabled by exposing a new capability.
  • oc: understand first how the CS3 APIs will fit this purpose. cern to create an issue to follow up.

Syncing unreadable folders

  • cern: oc expects a tree with read-write to be completely read-write, but sometimes there will be folders with more restrictive permissions, like read, therefore the sync client will abort the sync for the entire folder.

Disable user sharing for individual files

  • cern: we'd like to configure clients (mobile, desktop, web) to avoid creating shares to users on individual files.

AFAIK this is an eos limitation

  • oc: plan to do it by offering a new API that will be consumed by clients to render the correct dialogs for sharing.

AFAIK we want to strip down the sharing capabilities in the client and let it open the servers full page sharing dialog in the browser to manage more complicated permissions. This allows handling additional share types introduced by applications without changing client code.

Context menu for applications in desktop sync client

  • cern: the desktop sync client offers an OS-integrated context menu to share. We'd like to add our own entries to enable "open in Office" for example.
  • oc: won't add support for context menus. We'are going in the direction to open directly on the oc UI, for example, by clicking in the desktop show me versions, it will open a dialog with a browser inside rendering the versions tabs found in the oc UI. Idea is to extend this behaviour to applications, so "Open in Office" will open the file also in the browser.
    OCIS could provide views specially targeted to the desktop client making the integratin with the OS more native.

Old chunking algorithm

  • cern: still relying on old chunking.
  • oc: still in 2.7 but no support if something breaks.

is a matter of implementing it in the ocdav svc. do we want to use the direct up / download? we should, because then clients can directly talk to the data services, which, IMO, should use tus.io: Discussion in cs3org/reva#290

Request ID header

  • oc: sync client currently sends a request ID in the X-Request-Id header
  • cern: it can also be interesting if client could send extra information by using OpenCensus instrumentation (like start of the request seen from sync client, version number, ...) that will give us lot of insights from a client perspective rather than server-side.

UI Private link

  • cern: we'd like to allow people to just copy/paste the URL to access a shared resource.
  • oc: plan to merge private link and public link into one concept. Maier as product manager liked the idea of people just copy/pasting urls for sharing.

Configuration management for sync client: major priority for cern

  • cern: we want to dynamically provision users with configuration (hidden files, enabled features, ...). Currently we can configure at build time some constraints but that is not enough, we'd like to have it dynamically per user and easily changeable by and administrator without re-building a branded client and distributing it.
  • oc: current possibility is statically-branded builds with ownbrander and MDM-style deployment for mobile. There are some plans for user profiles but they will come with OCIS and new APIs that will allow us to achieve such functionality. Michael with follow up with configuration team as he wants to converge all configuration options into a single namespace to brand all clients the same way. Once that is there, the dynamic configuration will follow.

Recall of an user desktop sync client configuration

  • cern: we'd like to recall a user configuration as an administrator to not ask the user to find hidden special files (.sqlite, .ocsync, ...) to send into a ticket.
  • oc: we currently have a way where a user can be asked to create a debug archive that currently contains only the log information. cern will send list of files to be included in the archive. The generated archive (zip) is generated and the user chooses where to store it.
  • cern: it would be very helpful if the archive is stored automatically in the sync folder so the archive arrives to the administrator automatically.
  • oc: Hannha mentioned the possibily to put a magic file into the synced folder to trigger that behaviour. To be follow up in a subsequent meeting. The feature of pressing F12 to get the logs has been disabled, the user has now the power to generate such archive at prefered location.

Symlinks synchronization

  • cern: we'd like to have the option to also synchronize a symbolic link
  • oc: it will be possible with the integration of ocis and the cs3apis

Shared journal for desktop sync client

  • cern: the current sync client stores one sync journal (logs, sqlitedb) per sync folder pair, having one is desired for debugging done by administrators.
  • oc: the new archive can get all those files.
  • the log contains everything that is in the sqlite db. db not needed for debugging
  • sync pairs can be added to the log. see above
  • related to the 'make support super easy' epic above, cc @pmaier1
  • @pmaier1 needs issue: AFAICT the client VFS might have or need new log files to debug it
    • all vfs events are in the debug log as well

Add a new sync folder pair desktop sync client wizard

  • cern: the current wizard is confusing for users as EOS evaluates permissions at a folder level not in a hirarchy, this causes some propfind requests to fail.
  • oc: cern to send detailed issue.

Update of Windows client

  • cern: update to 2.5* requires moving first to 2.4.3
  • oc: make sure cern has autoupdater on latest version
  • @labkode in theory it would be possible to build a custom branded 2.6 client that includes the migration magic. effort would need to be estimated. that might allow skipping the migration to 2.4.3, which was the only client version that contains the logic. If the end user updates the client himself the client can directly be updated. Only the an autoupdater needs to go through v2.4 ... but since that is an autoupdatetr anyway ... that should not be an issue, right?

Virtual filesystem: major priority for cern

  • cern: we'd like that this feature reaches a production state.
  • oc: we have now a new version integrated natively with the Windows operative > system like Dropbox and Google Drive does using native "cloud" APIs. We plan to the same for Mac OS Catalina. For Linux OS there is no clear path yet but we also want to offer this integration. Suggest to try with 2.6.0-rc2 and give feedback.
  • cern : we'll start testing it.
  • cern: tests so far didn't manange to connect to the production cluster by current capabilities.

Sync client syncs only UTF-8

  • cern: sync client converts latin and other encodings back to utf-8
  • oc: the user will could be notified that the file has been re-encoded to utf-8

Mobile client developments

  • oc: working on a new IOS application with better integration thans to IOS "FileProvider". Android application is getting re-implemented with a new architecture.
  • cern: will send issues found by users to see if they have been addressed. A requested feature is to have automatic synced folders/files, like a desktop sync client in the mobile.
  • oc: we plan to provide this functionality by having the same sync algorithm on the client.
  • @labkode please submit separate issues

Collaboration on Phoenix

  • oc: the UX needs to be improved. Efforts were done on functionality first. Focusing on improving the interface now.
  • cern: we have the concept of project spaces, that are collaborative spaces where people can work together.
  • oc: we also want to introduce such concept as "space rooms"

related discussion:

  • cern: that is a great idea, however the APIs for Phoenix need to be flexible enough to allow adding arbitrary new storage spaces without having to hack code. For example we exposed at some point in the past an application called "Global EOS" that allowed access to arbitrary storage pools.
  • cern: we want to allow users to copy/paste the url to access shared spaces. The motivation behind is that all our end-user targeted storage systems (DFS, AFS, EOS) expose a global path that allows people to copy/paste a path and gets access to the resource. The Web UI masks this path to the root url (/) having no way to allow such functionality (the workaround was the private link). We'd like that Phoenix brings this possibility either as a configuration option or as main product feature. This changes the semantic (by path) and also exposes parent directory names. This may or may not be desired, depending on context. So a holistic product design needed here as well.
  • oc: Patrick as product manager liked that concept. To be followed up on a subsequent call.
  • @pmaier1 needs issue, related to "UI Private link" above
  • cern: the owncloud SDK and Phoenix should bring the possibility to configure the SDK with a different storage basename, currently all operations are mapped to "/", but we might want to send to "/eos/user/g/gonzalhu", that way the breadcrumbs for the user will be shown in the UI, that is a pre-requisite to allow copy/paste urls.
  • oc: Thomas and Vince understood the requirement and it should be possible to do so in the ownCloud SDK, they will follow up on a github issue. CERN will provide feedback by implementing the CS3APIs connector to the ownCloud SDK.
  • oc: some metadata attributes are prefixed with oc namespaces, like "https://owncloud/ns/fileid", how they will fit in the CS3APIs calls?
  • cern: the same prefix will be used in the CS3APIs when they are vendor-specific, so there should be no problem. For more general attributes we'll use some exisitng date dictionaries. cern to check with digital repositories at CERN for attribute classification.

Collaboration on OCIS/REVA

  • oc: released a first version of the OCIS setup where services are split in various repositories: ocis, ocis-webdav, ocis-phoenix, ocis-reva. Currenlty not using any micro-services framework. Will like to move to go-kit framewok as micro is becoming more comercial.
  • cern: agreed that the choice of framework for OCIS should be based on the sustenaibiliy of the framework for long term and avoiding projects led by single individuals.
  • cern: the current ocis setup is enough to run Reva alongside oc components as the connection between ocis-reva services and ocis-webdav is done by using the CS3 APIS. Current setup allows for evolution of Reva and OCIS independently while ensuring compatibily of using Ocis with any version of Reva. However, it could also be interesting to see if Reva could be backwards compatible with internal OCIs services (pkg commands).
  • oc: that could be possible if Reva uses the same framework and components as ocis. Thomas could prototype it.
  • cern: like the idea, however needs further discussion with cernbox team and follow up in a subsequent call. Worry that multiple repository approach will lead to code duplication and API checks are spread across services.
  • oc: code duplication will be there but relying on code generation should reduce the burden and for the API check a common library could be created.

AFAICT progress can be seen in these issues

Ore, more generally, in the OCIS sprint project

  • cern: would like to follow the approach done with Restic for documentation: code lives alongside documentation and changelog of the software also. Also we'd like to tag a version and to have an automatic release.
  • oc: should be also doable in Drone and using the calens tool.
  • cern: done, thanks to Thomas for instructions.

CS3 APIS

  • cern: would like the cs3 apis to be compiled and publish to various languages.
  • oc: should be possible to achieve that by using Drone as the CI/CD service.
  • cern: done, thanks to Thomas for providing us intructions on how to do the setup.
  • oc: would be nice to have a proposal system for the CS3 APIs. In Javascript world there is TC39 for such discussions.
  • cern: would be make a lot of sense once the CS3APIs gets to a stable state and start to be used elsewhere. The current contributor guidelines should suffice for until then
  • oc: not many documentation of the CS3APIS
  • cern: working on a white paper on the CS3APIs to present the on CHEP, further documentation will follow alongside Reva.

@DeepDiver1975
Copy link
Member 

* [ ]  @deepdiver can we use the roles api to forbid sharing files? or not announce the permission?

Yes - the roles api has the necessary flexibility.

@butonic butonic removed their assignment Nov 25, 2019
@labkode
Copy link
Contributor Author

labkode commented Jan 10, 2020

Related issue is owncloud/android#2787.
This again shows the incompatibilities between different clients (desktop, web, mobile) by having different implementations of the protocol. That is main reason why we are moving ahead with grpc and these type of issues will not appear.

@DeepDiver1975
Copy link
Member

and these type of issues will not appear.

As long as the code generators for different languages and/or platforms behave properly ;-)

@labkode
Copy link
Contributor Author

labkode commented Jan 10, 2020 •
edited
Loading

and these type of issues will not appear.

As long as the code generators for different languages and/or platforms behave properly ;-)

Of course, but if they do Google has a big problem ;)

@labkode labkode mentioned this issue Feb 11, 2020
Closed
@butonic butonic mentioned this issue Aug 21, 2020
Closed
refs added a commit that referenced this issue Sep 18, 2020
@refs
Merge pull request #43 from owncloud/release/v2.2.0
7d5e6c5 
Prepare 2.2.0 release
refs added a commit that referenced this issue Sep 18, 2020
@refs
Merge pull request #43 from owncloud/fix-middlware-cache-hit
9142f91
@tbsbdr tbsbdr mentioned this issue Feb 23, 2021
Open
@settings settings bot removed the p3-medium label Apr 7, 2021
@stale
Copy link

stale bot commented Jul 4, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Status:Stale label Jul 4, 2021
@stale stale bot closed this as completed Jul 14, 2021
@pascalwengerter
Copy link
Contributor

pascalwengerter commented Jul 15, 2021 •
edited
Loading

@labkode could you revisit this, split it up in actionable (separate) tickets and then close this issue? :)

@stale stale bot removed the Status:Stale label Jul 15, 2021
@stale
Copy link

stale bot commented Oct 30, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Status:Stale label Oct 30, 2021
@stale stale bot closed this as completed Nov 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Early-Adopter:CERN Type:Discussion
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@butonic @DeepDiver1975 @diocas @micbar @labkode @refs @pascalwengerter