Skip to content

Commit 1e19904

Browse files
aibaarsaibaars
authored
Merge pull request #134 from github/literals
Add and expand AST classes for literals
2 parents 3f4b4b3 + c019da8 commit 1e19904

32 files changed

Lines changed: 29700 additions & 24751 deletions

‎Cargo.lock‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎extractor/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ flate2 = "1.0"
1111
node-types = { path = "../node-types" }
1212
tree-sitter = "0.17"
1313
tree-sitter-embedded-template = { git = "https://github.com/aibaars/tree-sitter-embedded-template", rev = "d4aac29c08aa7c596633d00b5ec2dd2d247eafe4" }
14-
tree-sitter-ruby = { git = "https://github.com/tree-sitter/tree-sitter-ruby.git", rev = "add8cb36d5fc0a00d4499ba2e8eedc04a38a2488" }
14+
tree-sitter-ruby = { git = "https://github.com/tree-sitter/tree-sitter-ruby.git", rev = "2503f005d917c7aa4726dfe19398dc1a4a299d94" }
1515
clap = "2.33"
1616
tracing = "0.1"
1717
tracing-subscriber = { version = "0.2", features = ["env-filter"] }

‎generator/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,4 @@ edition = "2018"
1010
node-types = { path = "../node-types" }
1111
tracing = "0.1"
1212
tracing-subscriber = { version = "0.2", features = ["env-filter"] }
13-
tree-sitter-ruby = { git = "https://github.com/tree-sitter/tree-sitter-ruby.git", rev = "add8cb36d5fc0a00d4499ba2e8eedc04a38a2488" }
13+
tree-sitter-ruby = { git = "https://github.com/tree-sitter/tree-sitter-ruby.git", rev = "2503f005d917c7aa4726dfe19398dc1a4a299d94" }

‎prepare-db-upgrade.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Upgrading the Ruby database schema
2+
3+
The schema (`ql/src/ruby.dbscheme`) is automatically generated from tree-sitter's `node-types.json`. When the tree-sitter grammar changes, the database schema is likely to change as well, and we need to write an upgrade script. This document explains how to do that.
4+
5+
## Process Overview
6+
7+
1. Commit the change to `ruby.dbscheme` (along with any library updates required to work with the change).
8+
2. Run `prepare-db-upgrade.sh`.
9+
3. Fill in the details in `upgrade.properties`, and add any required upgrade queries.
10+
11+
It may be helpful to look at some of the existing upgrade scripts, to see how they work.
12+
13+
## Details
14+
15+
### Generating a Ruby database upgrade script
16+
17+
Schema changes need to be accompanied by scripts that allow us to upgrade databases that were generated with older versions of the tools, so they can use new query functionality (albeit with possibly degraded results).
18+
19+
#### The easy (mostly automatic) way
20+
21+
The easy way to generate an upgrade script is to run the `prepare-db-upgrade.sh` script. This will generate a skeleton upgrade directory, leaving you to fill out the details in the `upgrade.properties` file.
22+
23+
#### upgrade.properties
24+
25+
It will look something like:
26+
27+
```
28+
description: what it does
29+
compatibility: partial
30+
some_relation.rel: run some_relation.qlo
31+
```
32+
33+
The `description` field is a textual description of the aim of the upgrade.
34+
35+
The `compatibility` field takes one of four values:
36+
37+
* **full**: results from the upgraded snapshot will be identical to results from a snapshot built with the new version of the toolchain.
38+
39+
* **backwards**: the step is safe and preserves the meaning of the old database, but new features may not work correctly on the upgraded snapshot.
40+
41+
* **partial**: the step is safe and preserves the meaning of the old database, but you would get better results if you rebuilt the snapshot with the new version of the toolchain.
42+
43+
* **breaking**: the step is unsafe and will prevent certain queries from working.
44+
45+
The `some_relation.rel` line(s) are the actions required to do the database upgrade. Do a diff on the the new vs old `.dbscheme` file to get an idea of what they have to achieve. Sometimes you won't need any upgrade commands – this happens when the dbscheme has changed in "cosmetic" ways, for example by adding/removing comments or changing union type relationships, but still retains the same on-disk format for all tables; the purpose of the upgrade script is then to document the fact that it's safe to replace the old dbscheme with the new one).
46+
47+
Some typical upgrade commands look like this:
48+
49+
```
50+
// Delete a relation that has been replaced in the new scheme
51+
obsolete.rel: delete
52+
53+
// Create a new version of a table by applying a simple RA expression to an
54+
// existing table. The example duplicates the 'id' column of input.rel as
55+
// the last column of etended.rel, perhaps to record our best guess at
56+
// newly-populated "source declaration" information.
57+
extended.rel: reorder input.rel (int id, string name, int parent) id name parent id
58+
59+
// Create relationname.rel by running relationname.qlo and writing the query
60+
// results as a .rel file. The query file should be named relationname.ql and
61+
// should be placed in the upgrade directory. It should avoid using the default
62+
// QLL library, and will run in the context of the *old* dbscheme.
63+
relationname.rel: run relationname.qlo
64+
```
65+
66+
#### Testing your upgrade script
67+
68+
Upgrade scripts can be a little bit fiddly, so it's essential that you test them. You might do so as follows:
69+
70+
1. Create a snapshot of your favourite project using the old version of the code.
71+
72+
2. Switch to the new version of the code.
73+
74+
3. Try to run some queries that will depend on your upgrade script working correctly.
75+
76+
4. Observe the upgrade being performed in the query server log.
77+
78+
5. Verify that your queries produced sensible results.
79+
80+
#### Doing the upgrade manually
81+
82+
To create the upgrade directory manually, without using `prepare-db-upgrade.sh`:
83+
84+
1. Get a hash of the old `.dbscheme` file, from just before your changes. You can do this by checking out the code prior to your changes and running `git hash-object ql/src/ruby.dbscheme`
85+
86+
2. Go back to your branch and create an upgrade directory with that hash as its name, for example: `mkdir upgrades/454f1e15151422355049dc4f1f0486a03baeffef`
87+
88+
3. Copy the old `.dbscheme` file to that directory, using the name old.dbscheme.
89+
`cp ql/src/ruby.dbscheme upgrades/454f1e15151422355049dc4f1f0486a03baeffef/old.dbscheme`
90+
91+
4. Put a copy of your new `.dbscheme` file in that directory and create an `upgrade.properties` file (as described above).

‎prepare-db-upgrade.sh‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
#!/bin/sh
2+
#
3+
# Prepare the upgrade script directory for a Ruby database schema upgrade.
4+
5+
set -e
6+
set -u
7+
8+
app_name="$(basename "$0")"
9+
10+
usage()
11+
{
12+
exit_code="$1"
13+
shift
14+
15+
cat >&2 <<EOF
16+
${app_name}: $@
17+
${app_name}: Generate skeleton upgrade script.
18+
Usage: ${app_name} [--prev_hash <COMMITISH>]"
19+
20+
--prev-hash <COMMITISH>
21+
Hash/branch to use to get SHA1 for previous DB scheme.
22+
Default: origin/main
23+
24+
Must be run within the git repo needing an update.
25+
EOF
26+
exit "${exit_code}"
27+
}
28+
29+
prev_hash="origin/main"
30+
31+
while [ $# -gt 0 ]; do
32+
case "$1" in
33+
-x)
34+
set -x
35+
;;
36+
-h | --help)
37+
usage 0
38+
;;
39+
--prev-hash)
40+
if [ $# -eq 1 ]; then
41+
usage 2 "--prev-hash requires Commit/Branch option"
42+
fi
43+
shift
44+
prev_hash="$1"
45+
;;
46+
--)
47+
shift
48+
break
49+
;;
50+
-*)
51+
usage 2 "Unrecognised option: $1"
52+
;;
53+
*)
54+
break
55+
;;
56+
esac
57+
shift
58+
done
59+
60+
if [ $# -gt 0 ]; then
61+
usage 2 "Unrecognised operand: $1"
62+
fi
63+
64+
scheme_file="ql/src/ruby.dbscheme"
65+
upgrade_root="upgrades"
66+
67+
check_hash_valid()
68+
{
69+
if [ ${#2} -ne 40 ]; then
70+
echo "Did not get expected $1 hash: $2" >&2
71+
exit 2
72+
fi
73+
}
74+
75+
# Get the hash of the previous and current DB Schema files
76+
prev_hash="$(git show "${prev_hash}:${scheme_file}" | git hash-object --stdin)"
77+
check_hash_valid previous "${prev_hash}"
78+
current_hash="$(git hash-object "${scheme_file}")"
79+
check_hash_valid current "${current_hash}"
80+
if [ "${current_hash}" = "${prev_hash}" ]; then
81+
echo "No work to be done."
82+
exit
83+
fi
84+
85+
# Copy current and new dbscheme into the upgrade dir
86+
upgradedir="${upgrade_root}/${prev_hash}"
87+
mkdir -p "${upgradedir}"
88+
89+
cp "${scheme_file}" "${upgradedir}"
90+
git cat-file blob "${prev_hash}" > "${upgradedir}/old.dbscheme"
91+
92+
# Create the template upgrade.properties file.
93+
cat <<EOF > "${upgradedir}/upgrade.properties"
94+
description: <INSERT DESCRIPTION HERE>
95+
compatibility: full|backwards|partial|breaking
96+
EOF
97+
98+
# Tell user what we've done
99+
cat <<EOF
100+
Created upgrade directory here:
101+
${upgradedir}
102+
103+
Please update:
104+
${upgradedir}/upgrade.properties
105+
with appropriate upgrade instructions
106+
EOF

‎ql/src/codeql_ruby/AST.qll‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import ast.Call
33
import ast.Control
44
import ast.Constant
55
import ast.Expr
6+
import ast.Literal
67
import ast.Method
78
import ast.Module
89
import ast.Parameter

0 commit comments

Comments
 (0)