Skip to content

Commit c1112fb

Browse files
bestanderbestander
authored
Workspaces phase 3: linking (#66)
RFC for linking between workspaces
1 parent 572187b commit c1112fb

File tree

2 files changed

+201
-1
lines changed

2 files changed

+201
-1
lines changed

‎accepted/0000-workspaces-link-phase-3.md

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
- Start Date: 2017-05-18
2+
- RFC PR:
3+
- Yarn Issue: https://github.com/yarnpkg/yarn/issues/3294
4+
5+
# Yarn workspaces phase 3: linking workspaces to each other
6+
7+
## Summary
8+
9+
A continuation of https://github.com/yarnpkg/rfcs/pull/60.
10+
Ability for workspaces to refer each other when testing packages in integration.
11+
12+
## Motivation
13+
14+
People tend to split larger projects into self contained packages that are published to npm independently. The workspaces feature is being developed for Yarn to address this workflow.
15+
16+
In particular, testing packages that refer other packages from the same codebase can be difficult because Node.js and front end bundling tools would look up the referred packages in node_modules folder as it should be installed from npm registry.
17+
18+
Yarn Workspaces need to be able to refer to other local packages the same way when local packages are in development mode (source of truth is the package source code) and in production mode (source of truth is the package installed from npm).
19+
20+
## Detailed design
21+
22+
The structure of the source code is following
23+
24+
```
25+
| jest/
26+
| ---- package.json
27+
| ---- packages/
28+
| -------- jest-matcher-utils/
29+
| ------------ package.json
30+
| -------- jest-diff/
31+
| ------------ package.json
32+
...
33+
```
34+
35+
Top level package.json is like
36+
37+
```
38+
{
39+
"private": true,
40+
"name": "jest",
41+
"devDependencies": {
42+
},
43+
"workspaces": [
44+
"packages/*"
45+
]
46+
}
47+
```
48+
jest-matcher-utils (workspace referred by another one)
49+
```
50+
{
51+
"name": "jest-matcher-utils",
52+
"description": "...",
53+
"version": "20.0.3",
54+
"repository": {
55+
"type": "git",
56+
"url": "https://github.com/facebook/jest.git"
57+
},
58+
"license": "...",
59+
"main": "...",
60+
"browser": "...",
61+
"dependencies": {
62+
"chalk": "^1.1.3",
63+
"pretty-format": "^20.0.3"
64+
}
65+
}
66+
```
67+
68+
jest-diff (workspace that refers jest-matcher-utils)
69+
```
70+
{
71+
"name": "jest-diff",
72+
"version": "20.0.3",
73+
"repository": {
74+
"type": "git",
75+
"url": "https://github.com/facebook/jest.git"
76+
},
77+
"license": "...",
78+
"main": "...",
79+
"browser": "...",
80+
"dependencies": {
81+
"chalk": "^1.1.3",
82+
"diff": "^3.2.0",
83+
"**jest-matcher-utils**": "^20.0.3",
84+
"pretty-format": "^20.0.3"
85+
}
86+
}
87+
```
88+
89+
When user runs yarn install, this folder structure of the Workspace gets created
90+
```
91+
| jest/
92+
| ---- node_modules/
93+
| -------- chalk/
94+
| -------- diff/
95+
| -------- pretty-format/
96+
| ---- package.json
97+
| ---- packages/
98+
| -------- jest-matcher-utils/
99+
| ------------ node_modules/ (empty, all dependencies hoisted to the root)
100+
| ------------ package.json
101+
| -------- jest-diff/
102+
| ------------ node_modules/
103+
| ---------------- **jest-matcher-utils**/ (symlink) -> ../jest-matcher-utils
104+
| ------------ package.json
105+
...
106+
```
107+
108+
`jest/packages/jest-diff/node_modules/**jest-matcher-utils**` is a relative symlink to `jest/packages/jest-matcher-utils`
109+
110+
### Dependencies and version matching
111+
112+
Yarn would only link workspaces to each other if they match semver conditions.
113+
For example,
114+
115+
* `jest-matcher-utils` package.json is `20.0.3`
116+
* if `jest-diff` package.json dependencies has `jest-matcher-utils` with version specifier that matches 20.0.3, e.g. `"^20.0.3"` then Yarn will make a link from `jest-diff/node_modules/jest-matcher-utils` to `jest-matcher-utils` workspace
117+
* if `jest-diff` package.json dependencies has `jest-matcher-utils` with version specifier that does not match `20.0.3`, e.g. `"^19.0.0"` then Yarn would fetch `jest-matcher-utils@^19.0.0` from npm registry and install it the regular way
118+
119+
120+
### Problems with peer dependencies and hoisting
121+
122+
There is a common [peer dependency problem](http://codetunnel.io/you-can-finally-npm-link-packages-that-contain-peer-dependencies/) when using **yarn link** on local packages that people can work around in Node 6+ by setting **--preserve-symlinks** runtime flag.
123+
In Workspaces this situation won't be a problem because node_modules are installed in Workspace root and Node.js `require()` statements will resolve third-party peer dependencies by going up the folder tree and reaching the Workspaces' root node_modules.
124+
125+
As long as **jest-matcher-utils** does not make relative requires via its parent folder, flag **--preserve-symlinks** won't be necessary.
126+
127+
### Installing workspace in project root
128+
129+
Workspace root may also depend on a workspace and it should be installed the same way as other workspaces referring each other, e.g. if jest has `jest-matcher-utils` as dependency it will be installed
130+
131+
```
132+
| jest/
133+
| ---- node_modules/
134+
| -------- chalk/
135+
| -------- diff/
136+
| -------- pretty-format/
137+
| -------- **jest-matcher-utils**/ (symlink) -> ../packages/jest-matcher-utils
138+
| ---- package.json
139+
| ---- packages/
140+
| -------- jest-matcher-utils/
141+
| ------------ node_modules/ (empty, all dependencies hoisted to the root)
142+
| ------------ package.json
143+
...
144+
```
145+
146+
### Build scripts run order and cycle detection
147+
148+
From workspaces linking point of view installation phases look like this:
149+
150+
1. Resolution - Yarn identifies all workspaces and which workspaces refer each other
151+
2. Fetching - Yarn skips it for linked workspaces
152+
3. Linking - Yarn creates symlinks in node_modules of referring workspaces in the alphanumeric order of workspaces (starting with workspace root)
153+
4. Running scripts - Yarn runs (pre/post)install scripts for each linked workspace the same way it runs for packages from registry. Yarn already has a way to identify cycles between packages during this phase, in this case the order of execution is not controlled by user. To control scripts execution order for cycling dependencies there is an RFC gist https://gist.github.com/thejameskyle/abbc146a8cb5c26194c8acc4d14e7c30 by @thejameskyle
154+
155+
## Drawbacks
156+
157+
This solution creates a symlink inside node_modules of a Workspace package and symlinks have multiple drawbacks:
158+
159+
* Symlinks are not supported in all tools (e.g. watchman)
160+
* Symlinks are not supported well in all OS and environments (Windows pre 10 Creative updated, Docker on SMB storage(?))
161+
* A symlink to **jest-matcher-utils** does not emulate actual installation of the package, it just symlinks to the package source code - no prepublish and postinstall lifecycle scripts are executed and no files are filtered (as done during publishing)
162+
* A version change in package.json of **jest-matcher-utils** needs Yarn to rebuild the links, this may require file watching
163+
164+
## Alternatives
165+
166+
* Run **yarn pack** for **jest-matcher-utils** and install them from a .tgz file
167+
* PROS
168+
* Works without symlinks
169+
* Does not leak files from **jest-matcher-utils**, i.e. node_modules folder
170+
* Runs the same pack command as with real publishing to registry (tests folder and dev files won't be included)
171+
* CONS
172+
* Every file change during development of **jest-matcher-utils** will require Yarn to repack and install it
173+
* Pack/unpack is an excessive use of CPU
174+
* Hardlink files in **jest-matcher-utils** (only the ones listed for publishing) into jest-diff/node_modules/**jest-matcher-utils**. Similar idea was expressed in the knit RFC https://github.com/yarnpkg/rfcs/pull/41
175+
* PROS
176+
* Works without symlinks' drawbacks
177+
* Partially emulates published package by leaving out non publishable files, e.g. node_modules folder
178+
* Changes in the hardlinked files will be reflected in referring workspace node_modules
179+
* CONS
180+
* Hardlinks have limited support in Windows pre 10
181+
* When new files are created/removed in **jest-matcher-utils** the hardlinks need to be regenerated, that may require file watching to get good developer experience otherwise developer needs to run yarn install on every significant change
182+
* This does not simulate actual installation of the package as no prepublish and postinstall lifecycle scripts are executed
183+
184+
Yarn Workspaces could implement all of the above linking strategies and give developers a choice which one to choose for their project.
185+
Or the alternatives could be merged in a single solution for isolated e2e testing.
186+
187+
## Unresolved questions
188+
189+
* Is there an issue with Node resolving real paths in symlinked folders (https://github.com/nodejs/node/issues/3402) with this solution?
190+
If workspaces don't make relative requires outside of their root (e.g. a file in a sibling folder to the one with workspace's package.json), all requires should resolve the same way.
191+
192+
* Does it need to work for other type of packages: git, file, etc?
193+
194+
* As described in Workspace phase 1 RFC (https://github.com/yarnpkg/rfcs/pull/60) there is only one lockfile per workspace. Does yarn.lock need to reference that `jest-matcher-utils@^20.0.0` is resolved as a link to a folder?
195+
196+
* Combining multiple workspaces is out of scope of this document.
197+
198+
* (related to general Workspaces RFC) How do we prevent people from publishing package and forgetting to setup correct dependencies for every workspace? E.g. `left-pad` may be absent from a workspace package.json and be present in the workspace root package.json. Testing the workspace code with node_modules installed in the root won't reveal this issue.

‎accepted/0000-warkspaces-install.md renamed to ‎implemented/0000-workspaces-install-phase-1.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -157,4 +157,6 @@ Having multi-project dependency management natively in Yarn gives us a more cohe
157157

158158
# Unresolved questions
159159

160-
Running lifecycle scripts may cause unexpected results if they require a specific folder structure in node_modules.
160+
* Running lifecycle scripts may cause unexpected results if they require a specific folder structure in node_modules.
161+
162+
* How do we prevent people from publishing package and forgetting to setup correct dependencies for every workspace? E.g. `left-pad` may be absent from a workspace package.json and be present in the workspace root package.json. Testing the workspace code with node_modules installed in the root won't reveal this issue.

0 commit comments

Comments
 (0)