Dev/testing strategy using a matrix

[Jovonni]

No description provided.

[Copilot]

Pull request overview

This PR implements a comprehensive testing strategy using a matrix-based approach for cloud functions in a Kubernetes environment. The changes introduce centralized test orchestration, enhanced parallel execution capabilities, and support for multiple new function types including Rust, Python, and various integration patterns.

Changes:

• Centralized test runner with dynamic port allocation for parallel execution
• Matrix-based GitHub Actions workflow for testing individual functions
• New runtime support including Rust (actix-web), Python (PyTorch/Flask), and multiple language/framework combinations
• Standardized function structure with TypeScript compilation, GraphQL integration, and environment variable injection

Reviewed changes

Copilot reviewed 77 out of 81 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
scripts/test-runner.ts	Enhanced centralized test orchestration with dynamic port allocation, single function mode, and environment variable injection
scripts/run-function-test.sh	Shell script for invoking centralized test runner for individual functions
package.json	Added runtime dependencies (dotenv, jest, ts-jest) and build tools (tweetnacl, bs58 types)
k8s/base/functions/*.yaml	Knative service definitions for new functions (stripe, rust-hello-world, pytorch-gpu, opencode-headless, llm-internal-calvin, llm-external, github-repo-creator, crypto-login)
jest.config.js	Root-level Jest configuration for TypeScript support
functions/*/tsconfig.json	Standardized TypeScript configurations with declaration generation and proper exclusions
functions/*/src/index.ts	Refactored function implementations to use async handler pattern with GraphQL client context
functions/*/package.json	Updated package metadata with publishConfig, dependencies, and test scripts
functions/*/tests/index.test.ts	Integration tests orchestrating Kubernetes jobs with pod log verification
functions/_runtimes/node/runner.js	Universal Node.js runtime wrapper providing Express server, GraphQL client, and standardized request handling
functions/_runtimes/node/Dockerfile.test	Updated test image with pnpm-based installations
functions/test-utils.ts	Shared test utilities for job teardown with NO_TEARDOWN support
Makefile	Enhanced build targets with individual test shortcuts and cleanup commands
.github/workflows/test-k8s-deployment.yaml	Matrix-based workflow for parallel function testing

Comments suppressed due to low confidence (7)

scripts/test-runner.ts:1

• Duplicate comment on lines 7 and 9. Remove one of these identical comments.
  scripts/test-runner.ts:1
• Random port selection may still result in collisions if multiple runners start simultaneously. Consider using port 0 to let the OS assign an available port, or implement a port reservation mechanism with retries.
  scripts/test-runner.ts:1
• The -u flag (update snapshots) should not be used in CI/test runners as it automatically updates snapshots without validation. Remove this flag to ensure tests fail when snapshots don't match.
  functions/opencode-headless/_calvincode_build:1
• This appears to be a Git submodule reference file. According to .gitignore line 140, _calvincode_build should be ignored. This file should not be committed to the repository.
  functions/opencode-headless/scripts/build-calvin.sh:1
• Using hardcoded line numbers (81-95) for sed replacement is fragile and will break if the upstream build script changes. Consider using pattern-based replacement or version pinning with a documented rationale.
  functions/pytorch-gpu/Dockerfile:1
• This comment is unclear about the actual configuration. The function is named 'pytorch-gpu' but installs CPU-only PyTorch. Either rename the function to clarify it's for testing, or document that GPU support requires a different base image.
  functions/stripe-function/tests/index.test.ts:1
• The KubernetesClient is instantiated twice, with the first instance on line 15 being immediately overwritten on line 20. Remove the unused first instantiation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Hardcoding error message strings for status code determination is fragile. Consider having functions return an error object with a statusCode property, or use error types/classes for better maintainability.

Suggested change

	app.post('/', async (req, res) => {
	try {
	const result = await handler(req.body, { client, headers: req.headers });
	// Standard Shim Error Handling Heuristics
	if (result && result.error) {
	// Heuristics for 400 vs 500
	if (['Missing prompt', 'Unsupported provider', 'Missing "query" in payload',
	'Missing repoName or githubToken', 'Missing X-Database-Id header or DEFAULT_DATABASE_ID',
	'Missing required field', "Either 'html' or 'text' must be provided",
	"Missing address, message, or signature"].some(s => result.error.includes(s) || s === result.error)) {
	return res.status(400).json(result);
	}
	return res.status(500).json(result);
	const BAD_REQUEST_ERROR_MESSAGES = [
	'Missing prompt',
	'Unsupported provider',
	'Missing "query" in payload',
	'Missing repoName or githubToken',
	'Missing X-Database-Id header or DEFAULT_DATABASE_ID',
	'Missing required field',
	"Either 'html' or 'text' must be provided",
	'Missing address, message, or signature',
	];
	app.post('/', async (req, res) => {
	try {
	const result = await handler(req.body, { client, headers: req.headers });
	// Standard Shim Error Handling Heuristics
	if (result && result.error) {
	// Prefer structured statusCode information if available
	let statusCode;
	if (typeof result.statusCode === 'number') {
	statusCode = result.statusCode;
	} else if (typeof result.error === 'object' && result.error !== null && typeof result.error.statusCode === 'number') {
	statusCode = result.error.statusCode;
	}
	// Fallback to legacy message-based heuristics if no explicit statusCode is provided
	if (!statusCode) {
	if (typeof result.error === 'string' && BAD_REQUEST_ERROR_MESSAGES.some(s => result.error.includes(s) || s === result.error)) {
	statusCode = 400;
	} else {
	statusCode = 500;
	}
	}
	return res.status(statusCode).json(result);

[Copilot]

The matrix runs 14 functions sequentially in the same job with a 45-minute timeout. Consider adding a per-function timeout or splitting into separate jobs to prevent one slow/hanging function from blocking others.

[Copilot]

Code attempts to fetch logs before the job has been created (lines 48-68 appear before job creation at line 98). This will fail. Move the log fetching logic to after the job execution loop or remove this premature section.

Suggested change

	// 5. Fetch and Print Logs (Evidence)
	try {
	const podName = (await k8s.listCoreV1NamespacedPod({
	path: { namespace: NAMESPACE },
	query: { labelSelector: `job-name=${jobName}` }
	})).items[0].metadata.name;
	const res = await fetch(`http://127.0.0.1:8001/api/v1/namespaces/${NAMESPACE}/pods/${podName}/log`);
	const logs = await res.text();
	console.log('\n[Evidence] Function Pod Logs:\n' + logs + '\n');
	} catch (e) {
	console.warn("Failed to fetch logs for evidence", e);
	}
	// Cleanup
	try {
	await k8s.deleteBatchV1NamespacedJob({
	path: { namespace: NAMESPACE, name: jobName },
	query: { propagationPolicy: 'Background' }
	});
	} catch (e) { }