Skip to content

Commit 90b9078

Browse files
SasSwartSasSwart
committed
Merge remote-tracking branch 'origin/main' into jjs/prebuilds-user-group-membership
2 parents 4b63826 + 78af5e0 commit 90b9078

File tree

1,033 files changed

+56246
-34201
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,033 files changed

+56246
-34201
lines changed

‎.claude/docs/DATABASE.md

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
# Database Development Patterns
2+
3+
## Database Work Overview
4+
5+
### Database Generation Process
6+
7+
1. Modify SQL files in `coderd/database/queries/`
8+
2. Run `make gen`
9+
3. If errors about audit table, update `enterprise/audit/table.go`
10+
4. Run `make gen` again
11+
5. Run `make lint` to catch any remaining issues
12+
13+
## Migration Guidelines
14+
15+
### Creating Migration Files
16+
17+
**Location**: `coderd/database/migrations/`
18+
**Format**: `{number}_{description}.{up|down}.sql`
19+
20+
- Number must be unique and sequential
21+
- Always include both up and down migrations
22+
23+
### Helper Scripts
24+
25+
| Script | Purpose |
26+
|--------|---------|
27+
| `./coderd/database/migrations/create_migration.sh "migration name"` | Creates new migration files |
28+
| `./coderd/database/migrations/fix_migration_numbers.sh` | Renumbers migrations to avoid conflicts |
29+
| `./coderd/database/migrations/create_fixture.sh "fixture name"` | Creates test fixtures for migrations |
30+
31+
### Database Query Organization
32+
33+
- **MUST DO**: Any changes to database - adding queries, modifying queries should be done in the `coderd/database/queries/*.sql` files
34+
- **MUST DO**: Queries are grouped in files relating to context - e.g. `prebuilds.sql`, `users.sql`, `oauth2.sql`
35+
- After making changes to any `coderd/database/queries/*.sql` files you must run `make gen` to generate respective ORM changes
36+
37+
## Handling Nullable Fields
38+
39+
Use `sql.NullString`, `sql.NullBool`, etc. for optional database fields:
40+
41+
```go
42+
CodeChallenge: sql.NullString{
43+
String: params.codeChallenge,
44+
Valid: params.codeChallenge != "",
45+
}
46+
```
47+
48+
Set `.Valid = true` when providing values.
49+
50+
## Audit Table Updates
51+
52+
If adding fields to auditable types:
53+
54+
1. Update `enterprise/audit/table.go`
55+
2. Add each new field with appropriate action:
56+
- `ActionTrack`: Field should be tracked in audit logs
57+
- `ActionIgnore`: Field should be ignored in audit logs
58+
- `ActionSecret`: Field contains sensitive data
59+
3. Run `make gen` to verify no audit errors
60+
61+
## Database Architecture
62+
63+
### Core Components
64+
65+
- **PostgreSQL 13+** recommended for production
66+
- **Migrations** managed with `migrate`
67+
- **Database authorization** through `dbauthz` package
68+
69+
### Authorization Patterns
70+
71+
```go
72+
// Public endpoints needing system access (OAuth2 registration)
73+
app, err := api.Database.GetOAuth2ProviderAppByClientID(dbauthz.AsSystemRestricted(ctx), clientID)
74+
75+
// Authenticated endpoints with user context
76+
app, err := api.Database.GetOAuth2ProviderAppByClientID(ctx, clientID)
77+
78+
// System operations in middleware
79+
roles, err := db.GetAuthorizationUserRoles(dbauthz.AsSystemRestricted(ctx), userID)
80+
```
81+
82+
## Common Database Issues
83+
84+
### Migration Issues
85+
86+
1. **Migration conflicts**: Use `fix_migration_numbers.sh` to renumber
87+
2. **Missing down migration**: Always create both up and down files
88+
3. **Schema inconsistencies**: Verify against existing schema
89+
90+
### Field Handling Issues
91+
92+
1. **Nullable field errors**: Use `sql.Null*` types consistently
93+
2. **Missing audit entries**: Update `enterprise/audit/table.go`
94+
95+
### Query Issues
96+
97+
1. **Query organization**: Group related queries in appropriate files
98+
2. **Generated code errors**: Run `make gen` after query changes
99+
3. **Performance issues**: Add appropriate indexes in migrations
100+
101+
## Database Testing
102+
103+
### Test Database Setup
104+
105+
```go
106+
func TestDatabaseFunction(t *testing.T) {
107+
db := dbtestutil.NewDB(t)
108+
109+
// Test with real database
110+
result, err := db.GetSomething(ctx, param)
111+
require.NoError(t, err)
112+
require.Equal(t, expected, result)
113+
}
114+
```
115+
116+
## Best Practices
117+
118+
### Schema Design
119+
120+
1. **Use appropriate data types**: VARCHAR for strings, TIMESTAMP for times
121+
2. **Add constraints**: NOT NULL, UNIQUE, FOREIGN KEY as appropriate
122+
3. **Create indexes**: For frequently queried columns
123+
4. **Consider performance**: Normalize appropriately but avoid over-normalization
124+
125+
### Query Writing
126+
127+
1. **Use parameterized queries**: Prevent SQL injection
128+
2. **Handle errors appropriately**: Check for specific error types
129+
3. **Use transactions**: For related operations that must succeed together
130+
4. **Optimize queries**: Use EXPLAIN to understand query performance
131+
132+
### Migration Writing
133+
134+
1. **Make migrations reversible**: Always include down migration
135+
2. **Test migrations**: On copy of production data if possible
136+
3. **Keep migrations small**: One logical change per migration
137+
4. **Document complex changes**: Add comments explaining rationale
138+
139+
## Advanced Patterns
140+
141+
### Complex Queries
142+
143+
```sql
144+
-- Example: Complex join with aggregation
145+
SELECT
146+
u.id,
147+
u.username,
148+
COUNT(w.id) as workspace_count
149+
FROM users u
150+
LEFT JOIN workspaces w ON u.id = w.owner_id
151+
WHERE u.created_at > $1
152+
GROUP BY u.id, u.username
153+
ORDER BY workspace_count DESC;
154+
```
155+
156+
### Conditional Queries
157+
158+
```sql
159+
-- Example: Dynamic filtering
160+
SELECT * FROM oauth2_provider_apps
161+
WHERE
162+
($1::text IS NULL OR name ILIKE '%' || $1 || '%')
163+
AND ($2::uuid IS NULL OR organization_id = $2)
164+
ORDER BY created_at DESC;
165+
```
166+
167+
### Audit Patterns
168+
169+
```go
170+
// Example: Auditable database operation
171+
func (q *sqlQuerier) UpdateUser(ctx context.Context, arg UpdateUserParams) (User, error) {
172+
// Implementation here
173+
174+
// Audit the change
175+
if auditor := audit.FromContext(ctx); auditor != nil {
176+
auditor.Record(audit.UserUpdate{
177+
UserID: arg.ID,
178+
Old: oldUser,
179+
New: newUser,
180+
})
181+
}
182+
183+
return newUser, nil
184+
}
185+
```
186+
187+
## Debugging Database Issues
188+
189+
### Common Debug Commands
190+
191+
```bash
192+
# Check database connection
193+
make test-postgres
194+
195+
# Run specific database tests
196+
go test ./coderd/database/... -run TestSpecificFunction
197+
198+
# Check query generation
199+
make gen
200+
201+
# Verify audit table
202+
make lint
203+
```
204+
205+
### Debug Techniques
206+
207+
1. **Enable query logging**: Set appropriate log levels
208+
2. **Use database tools**: pgAdmin, psql for direct inspection
209+
3. **Check constraints**: UNIQUE, FOREIGN KEY violations
210+
4. **Analyze performance**: Use EXPLAIN ANALYZE for slow queries
211+
212+
### Troubleshooting Checklist
213+
214+
- [ ] Migration files exist (both up and down)
215+
- [ ] `make gen` run after query changes
216+
- [ ] Audit table updated for new fields
217+
- [ ] In-memory database implementations updated
218+
- [ ] Nullable fields use `sql.Null*` types
219+
- [ ] Authorization context appropriate for endpoint type

‎.claude/docs/OAUTH2.md

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
# OAuth2 Development Guide
2+
3+
## RFC Compliance Development
4+
5+
### Implementing Standard Protocols
6+
7+
When implementing standard protocols (OAuth2, OpenID Connect, etc.):
8+
9+
1. **Fetch and Analyze Official RFCs**:
10+
- Always read the actual RFC specifications before implementation
11+
- Use WebFetch tool to get current RFC content for compliance verification
12+
- Document RFC requirements in code comments
13+
14+
2. **Default Values Matter**:
15+
- Pay close attention to RFC-specified default values
16+
- Example: RFC 7591 specifies `client_secret_basic` as default, not `client_secret_post`
17+
- Ensure consistency between database migrations and application code
18+
19+
3. **Security Requirements**:
20+
- Follow RFC security considerations precisely
21+
- Example: RFC 7592 prohibits returning registration access tokens in GET responses
22+
- Implement proper error responses per protocol specifications
23+
24+
4. **Validation Compliance**:
25+
- Implement comprehensive validation per RFC requirements
26+
- Support protocol-specific features (e.g., custom schemes for native OAuth2 apps)
27+
- Test edge cases defined in specifications
28+
29+
## OAuth2 Provider Implementation
30+
31+
### OAuth2 Spec Compliance
32+
33+
1. **Follow RFC 6749 for token responses**
34+
- Use `expires_in` (seconds) not `expiry` (timestamp) in token responses
35+
- Return proper OAuth2 error format: `{"error": "code", "error_description": "details"}`
36+
37+
2. **Error Response Format**
38+
- Create OAuth2-compliant error responses for token endpoint
39+
- Use standard error codes: `invalid_client`, `invalid_grant`, `invalid_request`
40+
- Avoid generic error responses for OAuth2 endpoints
41+
42+
### PKCE Implementation
43+
44+
- Support both with and without PKCE for backward compatibility
45+
- Use S256 method for code challenge
46+
- Properly validate code_verifier against stored code_challenge
47+
48+
### UI Authorization Flow
49+
50+
- Use POST requests for consent, not GET with links
51+
- Avoid dependency on referer headers for security decisions
52+
- Support proper state parameter validation
53+
54+
### RFC 8707 Resource Indicators
55+
56+
- Store resource parameters in database for server-side validation (opaque tokens)
57+
- Validate resource consistency between authorization and token requests
58+
- Support audience validation in refresh token flows
59+
- Resource parameter is optional but must be consistent when provided
60+
61+
## OAuth2 Error Handling Pattern
62+
63+
```go
64+
// Define specific OAuth2 errors
65+
var (
66+
errInvalidPKCE = xerrors.New("invalid code_verifier")
67+
)
68+
69+
// Use OAuth2-compliant error responses
70+
type OAuth2Error struct {
71+
Error string `json:"error"`
72+
ErrorDescription string `json:"error_description,omitempty"`
73+
}
74+
75+
// Return proper OAuth2 errors
76+
if errors.Is(err, errInvalidPKCE) {
77+
writeOAuth2Error(ctx, rw, http.StatusBadRequest, "invalid_grant", "The PKCE code verifier is invalid")
78+
return
79+
}
80+
```
81+
82+
## Testing OAuth2 Features
83+
84+
### Test Scripts
85+
86+
Located in `./scripts/oauth2/`:
87+
88+
- `test-mcp-oauth2.sh` - Full automated test suite
89+
- `setup-test-app.sh` - Create test OAuth2 app
90+
- `cleanup-test-app.sh` - Remove test app
91+
- `generate-pkce.sh` - Generate PKCE parameters
92+
- `test-manual-flow.sh` - Manual browser testing
93+
94+
Always run the full test suite after OAuth2 changes:
95+
96+
```bash
97+
./scripts/oauth2/test-mcp-oauth2.sh
98+
```
99+
100+
### RFC Protocol Testing
101+
102+
1. **Compliance Test Coverage**:
103+
- Test all RFC-defined error codes and responses
104+
- Validate proper HTTP status codes for different scenarios
105+
- Test protocol-specific edge cases (URI formats, token formats, etc.)
106+
107+
2. **Security Boundary Testing**:
108+
- Test client isolation and privilege separation
109+
- Verify information disclosure protections
110+
- Test token security and proper invalidation
111+
112+
## Common OAuth2 Issues
113+
114+
1. **OAuth2 endpoints returning wrong error format** - Ensure OAuth2 endpoints return RFC 6749 compliant errors
115+
2. **Resource indicator validation failing** - Ensure database stores and retrieves resource parameters correctly
116+
3. **PKCE tests failing** - Verify both authorization code storage and token exchange handle PKCE fields
117+
4. **RFC compliance failures** - Verify against actual RFC specifications, not assumptions
118+
5. **Authorization context errors in public endpoints** - Use `dbauthz.AsSystemRestricted(ctx)` pattern
119+
6. **Default value mismatches** - Ensure database migrations match application code defaults
120+
7. **Bearer token authentication issues** - Check token extraction precedence and format validation
121+
8. **URI validation failures** - Support both standard schemes and custom schemes per protocol requirements
122+
123+
## Authorization Context Patterns
124+
125+
```go
126+
// Public endpoints needing system access (OAuth2 registration)
127+
app, err := api.Database.GetOAuth2ProviderAppByClientID(dbauthz.AsSystemRestricted(ctx), clientID)
128+
129+
// Authenticated endpoints with user context
130+
app, err := api.Database.GetOAuth2ProviderAppByClientID(ctx, clientID)
131+
132+
// System operations in middleware
133+
roles, err := db.GetAuthorizationUserRoles(dbauthz.AsSystemRestricted(ctx), userID)
134+
```
135+
136+
## OAuth2/Authentication Work Patterns
137+
138+
- Types go in `codersdk/oauth2.go` or similar
139+
- Handlers go in `coderd/oauth2.go` or `coderd/identityprovider/`
140+
- Database fields need migration + audit table updates
141+
- Always support backward compatibility
142+
143+
## Protocol Implementation Checklist
144+
145+
Before completing OAuth2 or authentication feature work:
146+
147+
- [ ] Verify RFC compliance by reading actual specifications
148+
- [ ] Implement proper error response formats per protocol
149+
- [ ] Add comprehensive validation for all protocol fields
150+
- [ ] Test security boundaries and token handling
151+
- [ ] Update RBAC permissions for new resources
152+
- [ ] Add audit logging support if applicable
153+
- [ ] Create database migrations with proper defaults
154+
- [ ] Update in-memory database implementations
155+
- [ ] Add comprehensive test coverage including edge cases
156+
- [ ] Verify linting compliance
157+
- [ ] Test both positive and negative scenarios
158+
- [ ] Document protocol-specific patterns and requirements

0 commit comments

Comments
 (0)