[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.
From: |
Pierre Langlois |
Subject: |
[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module. |
Date: |
Sun, 12 Feb 2023 12:24:33 +0000 |
User-agent: |
mu4e 1.8.13; emacs 28.2 |
Andrew Tropin <andrew@trop.in> writes:
> [[PGP Signed Part:Undecided]]
> On 2023-02-10 17:02, Pierre Langlois wrote:
>
>> Pierre Langlois <pierre.langlois@gmx.com> writes:
>>
>>> [[PGP Signed Part:Undecided]]
>>> Hi Andrew, thanks for pushing this along! It's great to see things
>>> getting merged.
>>>
>>> Andrew Tropin <andrew@trop.in> writes:
>>>
>>>> [[PGP Signed Part:Undecided]]
>>>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>>>
>>>>> On 2023-02-09 13:39, zimoun wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>>>
>>>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>>>
>>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>>>> right?
>>>>>>
>>>>>>
>>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>>
>>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>>
>>>>>> Leaving out all the others, right?
>>>>>
>>>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>>>> addresses some things from reviews and one commit, which adds html
>>>>> grammar.
>>>>>
>>>>> The html grammar is added for the testing purposes. It relies on
>>>>> generated parser.c and scanner.c and we will need to repackage it using
>>>>> grammar.js instead. I'm not sure if a separate build system is needed
>>>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>>>> which generates packages as in example with tree-sitter-grammar-html:
>>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>>>
>>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>>>> read it carefully, evaluate and either introduce new build system or
>>>>> just move all needed parts to tree-sitter-grammar function. WDYT?
>>>>> After we done with it we can package all other grammars.
>>>>
>>>> Ok, I realized that the proper build process for tree-sitter grammars is
>>>> a little harder than I expected, tree-sitter-build system make sense. I
>>>> reviewed it, made a small change:
>>>
>>> Ah great, I was going to comment to try and push for us to keep the
>>> build system. I originally went with a template package and inheritance,
>>> but Maxime suggested moving to a build-system which ended up making the
>>> package definitions a *lot* nicer IMO (see previous discussion here
>>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
>>> grammars that depend on each other more nicely I think.
>>>
>>>>
>>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>>> ;; Commentary:
>>>> ;;
>>>> ;; Build procedures for tree-sitter grammar packages. This is the
>>>> -;; builder-side code, which builds on top fo the node build-system.
>>>> +;; builder-side code, which builds on top of the node build-system.
>>>> ;;
>>>> ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>>> ;; shared object. The `tree-sitter generate' command invokes `node' in
>>>> order
>>>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>>> "-fno-exceptions"
>>>> "-O2"
>>>> "-g"
>>>> - "-o" ,(string-append lib "/" lang ".so")
>>>> + "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>>> ;; An additional `scanner.{c,cc}' file is sometimes
>>>> ;; provided.
>>>> ,@(cond
>>>>
>>>>
>>>> rewrote html grammar to use this build system and made it work with
>>>> built-in treesit package. Also, tried examples of c and cpp grammars
>>>> from patches in this thread.
>>>>
>>>> If you ok with it, I'll push the build system to master and update the
>>>> html grammar accordingly.
>>
>> Oh, I forgot to say, this change to the build system LGTM! I'm really
>> happy to see it merged soon :-). The path change will probably break the
>> emacs-28-based tree-sitter support, but that's OK, it's better for the
>> build-system to be made to target emacs 29's builtin support. I'm sure I
>> can work around for emacs 28.
>
> Actually, I think we can build grammars with both names, just providing
> two .so files instead of one. If you won't find a better workaround we
> can go this way.
That's no problem, I was able to work-around it quite easily. The
emacs-tree-sitter-langs package expects all the grammars supported to be
in a single directory, so we have to bundle them. So we can quite
easily create symlinks from the bundle with the expected names,
something like:
--8<---------------cut here---------------start------------->8---
(define-public emacs-tree-sitter-langs-grammar-bundle
(package
(name "emacs-tree-sitter-langs-grammar-bundle")
(source #f)
(version (package-version tree-sitter))
(build-system trivial-build-system)
(inputs
;; FIXME: Support for some languages is still left to package.
(list tree-sitter-bash
tree-sitter-c
tree-sitter-c-sharp
tree-sitter-cpp
tree-sitter-css
tree-sitter-elixir
tree-sitter-elm
tree-sitter-go
tree-sitter-haskell
tree-sitter-html
tree-sitter-java
tree-sitter-javascript
tree-sitter-json
tree-sitter-julia
tree-sitter-lua
tree-sitter-ocaml
tree-sitter-php
tree-sitter-python
tree-sitter-r
tree-sitter-rust
tree-sitter-ruby
tree-sitter-typescript))
(arguments
(list #:builder
(with-imported-modules '((guix build union)
(guix build utils))
#~(begin
(use-modules (ice-9 match)
(srfi srfi-1)
(guix build union)
(guix build utils))
(mkdir-p #$output)
(for-each
(lambda (lib)
(symlink lib
(string-append #$output "/"
(substring (basename lib)
(string-length
"libtree-sitter-")))))
(append-map (match-lambda
((name directory)
(find-files directory
"libtree-sitter-.*\\.so$")))
'#$(package-inputs this-package)))))))
(synopsis #f)
(description #f)
(home-page #f)
(license #f)))
--8<---------------cut here---------------end--------------->8---
Giving us:
--8<---------------cut here---------------start------------->8---
/gnu/store/fr9h59wgn55ilfifvm5df7xzxfwh38pc-emacs-tree-sitter-langs-grammar-bundle-0.20.7
├── bash.so ->
/gnu/store/9qzvcdlpryjl44klfq0i2liqf6lsp5jq-tree-sitter-bash-0.19.0/lib/tree-sitter/libtree-sitter-bash.so
├── cpp.so ->
/gnu/store/my02kq2dr6h48nmrl2dnfnm5rggx837w-tree-sitter-cpp-0.20.0-1.5ead1e2/lib/tree-sitter/libtree-sitter-cpp.so
├── c_sharp.so ->
/gnu/store/mp8jvbhx5xlgj5bxa52zhmk5x8i7md5h-tree-sitter-c-sharp-0.19.1/lib/tree-sitter/libtree-sitter-c_sharp.so
├── c.so ->
/gnu/store/y9ln76yx68q495vx8bnqdy87x7k8ihy5-tree-sitter-c-0.20.2/lib/tree-sitter/libtree-sitter-c.so
├── css.so ->
/gnu/store/fwji59sdwvqpqyb94m55qw1ak92cmys9-tree-sitter-css-0.19.0/lib/tree-sitter/libtree-sitter-css.so
├── elixir.so ->
/gnu/store/f6ismnxf7hlh1sq87zzarw56avvrzsl9-tree-sitter-elixir-0.19.0-1.de20391/lib/tree-sitter/libtree-sitter-elixir.so
├── elm.so ->
/gnu/store/56zjl2ljvba3z1j6zfmpfa8mq19a3g3a-tree-sitter-elm-5.6.3/lib/tree-sitter/libtree-sitter-elm.so
├── go.so ->
/gnu/store/qfy5sw6za96wkszbi21adxbxzggvjz3g-tree-sitter-go-0.19.1-1.05900fa/lib/tree-sitter/libtree-sitter-go.so
├── haskell.so ->
/gnu/store/cq3chmqcb8g1nf5mzb6yhzdd6x9gvg1v-tree-sitter-haskell-0.14.0-1.e30bdfd/lib/tree-sitter/libtree-sitter-haskell.so
├── html.so ->
/gnu/store/5pqfyjrg3yyvxaxidk690ffls3yb7wbi-tree-sitter-html-0.19.0/lib/tree-sitter/libtree-sitter-html.so
├── javascript.so ->
/gnu/store/28s09v3dfb1c9bdkq2791z9abxnih66p-tree-sitter-javascript-0.20.0/lib/tree-sitter/libtree-sitter-javascript.so
├── java.so ->
/gnu/store/i7jlqf5hbv0rhlkp4qlyc2d5ndw69dx4-tree-sitter-java-0.20.1/lib/tree-sitter/libtree-sitter-java.so
├── json.so ->
/gnu/store/5dv4r74p6gd9w2ncs6pjrhz5hbw47262-tree-sitter-json-0.19.0/lib/tree-sitter/libtree-sitter-json.so
├── julia.so ->
/gnu/store/5k37g1sdsllgh64p0w6ggabsni6jqlkr-tree-sitter-julia-0.19.0/lib/tree-sitter/libtree-sitter-julia.so
├── lua.so ->
/gnu/store/nqzn4a6kgb2rx4y44pxdm4sqf3pzcpz1-tree-sitter-lua-0.0.14/lib/tree-sitter/libtree-sitter-lua.so
├── ocaml_interface.so ->
/gnu/store/3h7krcj3xxclirb8afxh65ipabw1821l-tree-sitter-ocaml-0.19.0-1.0348562/lib/tree-sitter/libtree-sitter-ocaml_interface.so
├── ocaml.so ->
/gnu/store/3h7krcj3xxclirb8afxh65ipabw1821l-tree-sitter-ocaml-0.19.0-1.0348562/lib/tree-sitter/libtree-sitter-ocaml.so
├── php.so ->
/gnu/store/ymxf5m8jhihbrag7v2pghgydj3byp7wh-tree-sitter-php-0.19.0-1.435fa00/lib/tree-sitter/libtree-sitter-php.so
├── python.so ->
/gnu/store/ggmzicwfxb7gz1rr9lfkx8cak62bfw7v-tree-sitter-python-0.19.1-1.ed0fe62/lib/tree-sitter/libtree-sitter-python.so
├── r.so ->
/gnu/store/y9dxbnbb5dyf0rq3kpar7ip4w3lq6sb9-tree-sitter-r-0.0.1-1.80efda5/lib/tree-sitter/libtree-sitter-r.so
├── ruby.so ->
/gnu/store/ky8n30dw16ck6byaqnhbf9ib7xp7j0yw-tree-sitter-ruby-0.20.0/lib/tree-sitter/libtree-sitter-ruby.so
├── rust.so ->
/gnu/store/25zdpwrgq1xibhv7xpg64i4g71xah6g1-tree-sitter-rust-0.20.1/lib/tree-sitter/libtree-sitter-rust.so
├── tsx.so ->
/gnu/store/wqpcphz855yjrginwqrymd3xzzxb8k8l-tree-sitter-typescript-0.20.1/lib/tree-sitter/libtree-sitter-tsx.so
└── typescript.so ->
/gnu/store/wqpcphz855yjrginwqrymd3xzzxb8k8l-tree-sitter-typescript-0.20.1/lib/tree-sitter/libtree-sitter-typescript.so
--8<---------------cut here---------------end--------------->8---
signature.asc
Description: PGP signature
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., Andrew Tropin, 2023/02/09
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., zimoun, 2023/02/09
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., Andrew Tropin, 2023/02/12
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., Pierre Langlois, 2023/02/12
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., Andrew Tropin, 2023/02/14
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., Pierre Langlois, 2023/02/17
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., Pierre Langlois, 2023/02/10
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., Andrew Tropin, 2023/02/12
- [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module., Pierre Langlois, 2023/02/12