Skip to content

rework yjs-store-db syncing #205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 10, 2023
Merged

rework yjs-store-db syncing #205

merged 3 commits into from
Jan 10, 2023

Conversation

lihebi
Copy link
Collaborator

@lihebi lihebi commented Jan 9, 2023

This PR is a major rework of the syncing logic among Yjs, Zustand store, and DB. The syncing logic will be detailed in the comment below.

Additional changes:

  • Remove addPod graphQL API. Use the updatePod API instead, which will create a pod if it does not exist.
    • This fixes the pod creation & updates race condition. We don't have to await the pod creation before updating it in the front-end canvas.
    • Additional note: the addPod Zustand store action will not accept ApolloClient anymore. Instead, you can set the pod.dirty=true flag to trigger the periodical updates to DB.
  • use @reactflow/node-resizer for scope resize.

@lihebi
Copy link
Collaborator Author

lihebi commented Jan 10, 2023

The three data stores and their roles:

  • Yjs data: Yjs is the single source of truth in terms of canvas operations. It is in store.ydoc.nodesMap. This data is shared by all peers. The canvas is rendered from this data.
  • DB data: On canvas loading, if Yjs data is empty or inconsistent with the database, overwrite Yjs data with database data. That is, the database is the single source of truth in terms of eventual consistency.
  • Zustand store: The local Zustand store is used to store graph structure and metadata, as well as handle DB writes.

@lihebi
Copy link
Collaborator Author

lihebi commented Jan 10, 2023

The files: the main logic is implemented in three files:

  • lib/nodes.tsx: this implements Yjs observer
  • components/Canvas.tsx: implements the Canvas React component.
  • lib/store/canvasSlices.tsx: actually implements all the operations (updateView, copy/paste/cut logic)

@lihebi
Copy link
Collaborator Author

lihebi commented Jan 10, 2023

How the canvas nodes are rendered?

The ReactFlow canvas data is stored in store.nodes, and passed to <ReactFlow nodes={store.nodes}/>. Updating this state will trigger a canvas re-rendering. Only one action is allowed to actually update the nodes, the updateView function. Other actions are supposed to call it whenever a re-rendering is desired.

The updateView action computes the new nodes from the following data sources:

  • All nodes in Yjs nodesMap.
  • The pasting/cutting node. Note: these temporary nodes are maintained outside nodesMap because they are local to the local user.

Additional logic is implemented in updateView to modify the nodes, e.g.,

  • add background color according to scope level;
  • set the className for cutting nodes and active scope.

@lihebi
Copy link
Collaborator Author

lihebi commented Jan 10, 2023

Handling the changes made by remote peers

All changes made by remote peers are observed with useYjsObserver(). These observed changes (add,remove,update) will

  1. Write to the local Zustand store without DB sync.
  2. trigger updateView action to update the canvas.

@lihebi
Copy link
Collaborator Author

lihebi commented Jan 10, 2023

Handling the changes made by the local user

Overall, the local user's changes will

  1. write to nodesMap to sync with remote peer
  2. write to zustand store, sync with DB (using the dirty=true flag), then call updateView to update canvas.

The specific logic for each Canvas operation:

  • Add node:
    1. Update nodesMap to sync with the remote peer.
    2. Do local updates (zustand store, db, updateView)
  • Move a node: onNodesChange will observe the change. It calls applyChange to compute new nodes, and do:
    1. Set nodesMap to sync with the remote peer
    2. Do local updates (zustand store, db, updateView)
    3. Check if this is moved into scope. (handled by <ReactFlow onNodeDragStop={...}>)
  • Select nodes: onNodesChange observe the selection event, and record and update the local canvas.
  • Remove node:
    1. nodesMap.delete to sync with peer
    2. Do local updates (zustand store, db, updateView)
  • Paste
    • Begin (triggered by ctrl-v)
      • Create and set store.pastingNode. Set store.isPasting flag.
    • onMove: The "move" event listener will set store.pastingNode's position and update the view so that the pod will follow the cursor.
    • End:
      1. Update nodesMap to sync with the remote peer
      2. Do local updates (zustand store, db, updateView)
      3. Check if dropped into a scope. If so, call store.moveIntoScope action to update parentNode and recompute all scopes' levels.
    • Cancel: clear the pastingNode and isPasting
  • Cut (triggered by clicking the cut button): similar to pasting.

@lihebi lihebi merged commit 8b1f60a into main Jan 10, 2023
@lihebi lihebi deleted the rework-yjs-store-db-sync branch June 30, 2023 16:30
@lihebi lihebi mentioned this pull request Aug 10, 2023
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant