bug#35739: Bad signature from GNU ELPA

[Stefan Monnier]

> Not in detail, it's not an area of expertise of mine. We call
> (decode-coding-region (point-min) (point-max) 'undecided) on the
> payload of "https://elpa.gnu.org/packages/archive-contents";,
> which is raw text. The resulting buffer's buffer-file-coding-
> system is iso-latin-1-dos.

Indeed, it seems that url-insert-file-contents sets
buffer-file-coding-system.  Maybe we can use that in the emacs-26
branch.  Can you try the emacs-26 code with the patch below to see if it
fixes the current problem?


        Stefan


2019-05-29  Stefan Monnier  <monnier@iro.umontreal.ca>

        * lisp/emacs-lisp/package.el: Obey buffer-file-coding-system.
        `url-insert-file-contents` saves in buffer-file-coding-system
        the coding-system used to decode the contents.  Preserve this
        as the contents is moved from buffer to string to buffer, and use
        it when saving the contents to file, so as to try and better preserve
        the original byte sequence (bug#35739).
        (package--buffer-string, package--cs): New functions.
        (package--check-signature): Encode `string` if a coding-system
        was specified in buffer-file-coding-system.
        (package--download-one-archive, package-install-from-archive):
        Obey and preserve the buffer-file-coding-system if specified.


diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 1a185de4a5..46f7c91272 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1241,6 +1241,17 @@ package--check-signature-content
         (signal 'bad-signature (list sig-file)))
       good-signatures)))
 
+(defun package--buffer-string ()
+  (let ((string (buffer-string)))
+    (when (and buffer-file-coding-system
+               (> (length string) 0))
+      (put-text-property 0 1 'package--cs buffer-file-coding-system string))
+    string))
+
+(defun package--cs (string)
+  (and (> (length string) 0)
+       (get-text-property 0 'package--cs string)))
+
 (defun package--check-signature (location file &optional string async callback 
unwind)
   "Check signature of the current buffer.
 Download the signature file from LOCATION by appending \".sig\"
@@ -1260,8 +1271,12 @@ package--check-signature
 
 UNWIND, if provided, is a function to be called after everything
 else, even if an error is signaled."
-  (let ((sig-file (concat file ".sig"))
-        (string (or string (buffer-string))))
+  (let* ((sig-file (concat file ".sig"))
+         (string (or string (package--buffer-string)))
+         (cs (package--cs string)))
+    ;; Re-encode the downloaded file with the coding-system with which
+    ;; it was decoded, so we (hopefully) get the exact same bytes back.
+    (when cs (setq string (encode-coding-string string cs)))
     (package--with-response-buffer location :file sig-file
       :async async :noerror t
       ;; Connection error is assumed to mean "no sig-file".
@@ -1529,7 +1544,7 @@ package--download-one-archive
     :error-form (package--update-downloads-in-progress archive)
     (let* ((location (cdr archive))
            (name (car archive))
-           (content (buffer-string))
+           (content (package--buffer-string))
            (dir (expand-file-name (format "archives/%s" name) 
package-user-dir))
            (local-file (expand-file-name file dir)))
       (when (listp (read content))
@@ -1538,7 +1553,8 @@ package--download-one-archive
                 (member name package-unsigned-archives))
             ;; If we don't care about the signature, save the file and
             ;; we're done.
-            (progn (let ((coding-system-for-write 'utf-8))
+            (progn (let ((coding-system-for-write
+                          (or (package--cs content) 'utf-8)))
                      (write-region content nil local-file nil 'silent))
                    (package--update-downloads-in-progress archive))
           ;; If we care, check it (perhaps async) and *then* write the file.
@@ -1546,7 +1562,7 @@ package--download-one-archive
            location file content async
            ;; This function will be called after signature checking.
            (lambda (&optional good-sigs)
-             (let ((coding-system-for-write 'utf-8))
+             (let ((coding-system-for-write (or (package--cs content) 'utf-8)))
                (write-region content nil local-file nil 'silent))
              ;; Write out good signatures into archive-contents.signed file.
              (when good-sigs
@@ -1838,15 +1854,17 @@ package-install-from-archive
           (let ((save-silently t))
             (package-unpack pkg-desc))
         ;; If we care, check it and *then* write the file.
-        (let ((content (buffer-string)))
+        (let ((content (package--buffer-string)))
           (package--check-signature
            location file content nil
            ;; This function will be called after signature checking.
            (lambda (&optional good-sigs)
              ;; Signature checked, unpack now.
-             (with-temp-buffer (insert content)
-                               (let ((save-silently t))
-                                 (package-unpack pkg-desc)))
+             (with-temp-buffer
+               (insert content)
+               (setq buffer-file-coding-system (package--cs content))
+               (let ((save-silently t))
+                 (package-unpack pkg-desc)))
              ;; Here the package has been installed successfully, mark it as
              ;; signed if appropriate.
              (when good-sigs