How to avoid breaking pagination in Flask while filtering?

Submitted 3 years, 5 months ago
Ticket #261
Views 805
Language/Framework Python
Priority Medium
Status Closed

What can I do when my pagination breaks after first page when I specify the keyword in a form for search results? When I'am moving over pages without any filters, everything works perfectly fine. I had the same problem in my Django project, but I fixed it by creating a custom template tag, it was hard tho and now I can't manage to find any solution for my Flask App. Any ideas? Maybe modifying url?  First page is filtered correctly, even pages are counted properly - but when I swich on second page everything breaks... Please, it's urgent! I've been struggling with this for past few days!

1st page when "jquery" keyword is specified:

1st page

2nd page - completly breaks pagination and search result and backs to normal unfiltered query:

2nd page

VIEWS.PY

@app.route('/cve_list', methods=['GET', 'POST'])
def cve_list():
    search_form = SearchForm()
    page = request.args.get('page', 1, type=int)
    cve = CVE.query.order_by(CVE.date.desc()).paginate(page=page, per_page=2)

    if search_form.validate_on_submit():  # = when search_form was submitted
        form_data = search_form.search.data
        query = CVE.query.filter(CVE.content.contains(f'{form_data}')).paginate(page=page, per_page=2)
        cve_amount = CVE.query.filter(CVE.content.contains(f'{form_data}')).count()
        if cve_amount > 0:
            flash(f'Found {cve_amount} vulnerabilities based on your search - {form_data}', 'success')
        else:
            flash(f'No vulnerabilities found based on your search - {form_data}', 'warning')

        return render_template('cve_list.html',
                               title=f'Cve{form_data}',
                               form=search_form,
                               cve=query,
                               cve_amount=cve_amount)
    return render_template('cve_list.html',
                           title='Cve List',
                           form=search_form,
                           cve=cve)

HTML FILE

 {% for cve in cve.items %}
               <article class="media content-section">
                  <div class="media-body">
                     <div class="article-metadata">
                        <a class="mr-2">{{ cve.title }}</a>
                        <small class="text-muted">{{ cve.date }}</small>
                     </div>
                     <p class="article-content">{{ cve.content }}</p>
                  </div>
               </article>
               {% endfor %}

                <!-- PAGINATION -->
              {% for page_num in cve.iter_pages(left_edge=1, right_edge=1, left_current=1, right_current=2) %}
                    {% if page_num %}
                        {% if cve.page == page_num %}
                            <a class="btn btn-info mb-4" href="{{ url_for('cve_list', page=page_num) }}">{{ page_num }}</a>
                        {% else %}
                            <a class="btn btn-outline-info mb-4" href="{{ url_for('cve_list', page=page_num) }}">{{ page_num }}                                              </a>
                        {% endif %}
                    {% else %}
                    {% endif %}
                {% endfor %}
          <!-- PAGINATION -->

FORMS.PY

class SearchForm(FlaskForm):
    search = StringField('Enter keyword',
                         validators=[DataRequired(), Length(min=1, max=20)])
    submit = SubmitField('search')

FORM IN HTML

<!-- FORM SECTION -->
<div class="col-md-4">
    <div class="search-bar-section">
        <form method="GET" action="">
            {{ form.hidden_tag() }}
            <fieldset class="form-group">
                <div class="form-group">
                    {{ form.search.label(class="form-control-label") }} {% if form.search.errors %} 
                    {{ form.search(class="form-control form-control-lg is-invalid") }}
                    <div class="invalid-feedback">
                        {% for error in form.search.errors %}
                        <span>{{ error }}</span>
                        {% endfor %}
                    </div>
                    {% else %} {{ form.search(class="form-control form-control-lg") }} {% endif %}
                </div>
                <div class="form-group">
                    {{ form.submit(class="btn btn-light") }}
                </div>
            </fieldset>
        </form>
    </div>
</div>
<!-- FORM SECTION -->

URL on 1st page after submiting form with "jquery" keyword:  http://127.0.0.1:5000/cve_list

URL on 2nd page after the first one after submitting form: http://127.0.0.1:5000/cve_list?page=2

I don't know, urls in Django were much better. That's why I could solve the problem with your proposed method back then. It looks like Flask just don't track any changes to url or something, there's no even specified a keyword "jquery" in url. I don't get it... :/ Im so confused.

Submitted on Oct 23, 20

I am not sure any Flask Techions available in this forum. I will check this issue and get back to you. - Vengat 3 years, 5 months ago

can you show your url in second page & first page? - Vengat 3 years, 5 months ago
add a comment

2 Answers

Verified

I think lets try this approach, if doesnt work let me know , we can extend the logic,

Context Processors

Create this context processor in you rapp,

@app.context_processor
def utility_processor():
	def paginate_url(field_name, value, urlencode=None):
		get_query = f'{field_name}={value}'
		if urlencode:
			qs = urlencode.split('&')
			_filtered = filter(lambda p: p.split('=')[0] != field_name, qs)
			querystring = '&'.join(_filtered)
			get_query = f'{get_query}&{querystring}'
		return get_query

Now we can use this paginate_url function as a variable in our template like below,

{{ paginate_url('page', 1, urlencode value) }}

We need to figure out how to pass the urlencode from html to this context processor, then i think it should pretty much equivalent to Django template tags. May be try this, i will also explore more on this.

Submitted 3 years, 5 months ago

Hi arun! Thank you for your response. I actually fixed familiar issue using this method by in Django, not Flask. I will edit my post and add urls you asked for.

- hijaw39777 3 years, 5 months ago

Can you try this context processor approach?

- Vengat 3 years, 5 months ago

Check edited post, I don't think this will bring any results with these urls.

- hijaw39777 3 years, 5 months ago

actually we need to set the url like http://localhost:8000/?page=2&filter=jquery but i dont see query parameter string

- Vengat 3 years, 5 months ago

Yes you're absolutely right but how can I do that? I think that was done automaticly in django, but here... ehh. I really appreciate your support and time that you're putting into this. Thank you from the bottom of my heart, you're life-saver.

- hijaw39777 3 years, 5 months ago

No worries. How about using Flask restless Flask Restless option. Seems like it has context processor & query parameter options too.

- Vengat 3 years, 5 months ago

But what about using url_for function to generate url? Any ideas?

- hijaw39777 3 years, 5 months ago

Other option try this webargs. We can check the args here and you can keep your pagination as it is but on top of your route you can add it i guess.

- Vengat 3 years, 5 months ago

Ok, thats so much to do so I have to work on it a little bit. Can we continue this thread tomorrow? About 19:00 CEST?

- hijaw39777 3 years, 5 months ago

Sure

- Vengat 3 years, 5 months ago

Hi arun, check my comment to this post, it's getting even more weird..

- hijaw39777 3 years, 5 months ago

I didn't manage to fix this but I found something strange and interesting at once.

I've changed my code to this below and changed HTML form method to GET.

if request.method == 'GET':
    form_data = search_form.search.data

And you know what? Pagination works for the NONE result (I think it means that the search bar is empty) BUT.... it looks like this if statement is being triggered right after entering this page. At this point:

  • pagination works BUT without even specifying anything
  • search bar doesn't work at all
  • 'None' is paginated because the form is empty i guess

Check the pictures:

And one more thing, this pagination can work for every keyword I want to but I have to use it as a variable like so:

if request.method == 'GET':
    form_data = 'jquery'

And it is triggered right after entering this page, without even showing original query which suppose show all CVE codes. I am actually much more confused than before. It just means that pagination works but it somehow doesn't cooperate with the rest of the code. Im missing something so hard... maybe some mistakes in code but damn, I don't know what.. :/ 

Any ideas?

Submitted 3 years, 5 months ago

why cant you change to POST form request like this
form = SearchForm() if form.validate_on_submit(): search_term = form.query.data results = CVE.query.filter()

- Vengat 3 years, 5 months ago

In that case, I cannot use form.validate_on_submit() because it checks whether HTTP method is POST and won't trigger if it's GET (GET is used in my HTML right now). And if I will change to validate_on_submit, then I will have to switch to POST method in HTML and it will just be the same code and situation like before. I hope you understand me.

- hijaw39777 3 years, 5 months ago

In earlier did you use POST in your form?

- Vengat 3 years, 5 months ago

I think now we are more complicated the code, so can you reiterate after you changed the code pagination is working for filter?

- Vengat 3 years, 5 months ago

Yes, I'll paste the form code in my original post here.

- hijaw39777 3 years, 5 months ago

The code and screenshots you can see in my answer (and form code in original question) to this thread is this what I actually have right now. It's hard to explain what's going on right now to be honest. New screenshots in my answer are showing you what's happening RIGHT AFTER entering this page - as I tried to explain, it looks like the form is being submitted automaticly after entering page that's why it's querying db for "None" keyword because the form is empty. I wish I could chat with you in some other way like discord or something because it's hard to cooperate here.

- hijaw39777 3 years, 5 months ago

I would say just change that form to POST, yes we want to stick with this platform, I have submitted suggesttion to add chat app 😀

- Vengat 3 years, 5 months ago

When I'm changing form to POST: - original query works - searching bar works - form submitting works - even counting pages for that specific keyword works But unfortunately when I change page - pagination goes back to original query. It looks like the app forgets about the keyword after every time I change page after form submitting. * chat app is something that stack overflow is missing

- hijaw39777 3 years, 5 months ago

Yes but this is ticketing platform so don't worry as a SME ,I will follow up until your issue has been resolved, regarding GET request by default GET request will trigger thats why you are getting None even if you are not submitting, so may be we can add request.args.search value in your get request and make if its not none in your GET request and in your html template you can add if else template for this search parameter . Is it make sense?

- Vengat 3 years, 5 months ago

Yes. that might be an idea. But I'd highly appreciate if you could provide me more information or maybe some code on pastebin. So now we gonna try to change everything to GET request right? Includes HTML form and methods in my app.route, am I right? And thank you for what you said about following up until resolving issue, it really motivates, especially when it comes to working with somebody such experienced in this industry.

!!! also - would you like me to share a github repo with whole app so you could check a few things?

- hijaw39777 3 years, 5 months ago

Yes GET request only. That is the idea of this platform. Basically close the gap between the Developers. Its not like Q&A site, where people doesnt care about developers whether its resolved or not. And demotivating with negative votes n all. Here idea is community should help users/developers if they dont help, then our SME's roles is follow up with user and close the issue. Regarding, code , I need to prepare some skeleton.

- Vengat 3 years, 5 months ago

Ok, so I will try to make some changes by following your guidelines and you will tell me when I can send you a github repo. Is that OK for you? So I will change everything to GET, get rid of validate_on_submit - because it won't cooperate with our GET request and try request.args.search as you mentioned. But I can't understand that one thing you said: "and in your html template you can add if else template for this search parameter" You mean the jinja if, else right? Like: {{ if form.submit etc etc }} then {{ .... }}?

- hijaw39777 3 years, 5 months ago

Yes send me the git repo, request.args like when we submit the search button then grab that value if there is no value submitted then it will send None right then in our html file add jinja logic if search then show endif something like this

- Vengat 3 years, 5 months ago

Can you give me your github account link? I'll add you as a contributor to this repo. I prepared for you a simple script to send some real data to sqlite that im working on. If you want to, use it :) - it's in extras.py. Also this repo is before the GET, args changes that we decide to make. I want to kinda sink into and work on it for a few hours. I feel like I also need more knowledge about HTTP, I hope you understand. I will add you to this repo right after you will give me your account link. If thats not ok for you, i'll make it public. :-)

- hijaw39777 3 years, 5 months ago

here you go github

- Vengat 3 years, 5 months ago

I added you sir :-) Also another question, did you create this platform? :O

- hijaw39777 3 years, 5 months ago

No, I m Django SME in Teckiy. Do you like this platform 👋?

- Vengat 3 years, 5 months ago

Of course I do. Especially when somebody like you helps juniors or beginners like me. And this is not just a help based on one response. You're cooperating with me, and that's awesome. After I land my first job (now im 17 years old) I'll try to support this platform for sure. I invited you to repo, can you see it?

- hijaw39777 3 years, 5 months ago

Cool. I will check and let you know. Meanwhile if you do progress then keep me posted.

- Vengat 3 years, 5 months ago

May be try this, before change the form into GET request in your search form, def cve_list(): search_form = SearchForm(request.args) page = request.args.get('page', 1, type=int) cve = CVE.query.order_by(CVE.date.desc()).paginate(page=page, per_page=5) if search_form.validate(): form_data = search_form.search.data query = CVE.query.filter(CVE.content.contains(f'{form_data}')).order_by(CVE.date.desc()).paginate(page,per_page,error_out=False) paginated_query = query.paginate(page=page, per_page=2)

- Vengat 3 years, 5 months ago

Hi arun, I had much to do today so I couldn't work as much as I wanted to. I'll text you tomorrow at 20 CET and inform you about next steps. Have a good day!

- hijaw39777 3 years, 5 months ago

Hi, I tried that code you send me here. It doesn't work :/ It works almost the same as mine before, pagination still breaks. Now I'm changing everything to GET method. It's getting harder and harder.. Have you tried to run this app on your machine from github :)?

- hijaw39777 3 years, 5 months ago

I am unable to run that code in my local. Can you just print statement on request.args on each page request and check what data is rendering over there?

- Vengat 3 years, 5 months ago

I don't think I understand what you're trying to tell me :/ What do you exactly want me to do? Use code that you send me? Code with POST or code with GET?

- hijaw39777 3 years, 5 months ago

use the code which i sent recently and add print statement on page variable and filter variable and see what you are getting when you do pagination. Basically want to see whether the filter is carried over or not.

- Vengat 3 years, 5 months ago

With code that you provided search bar doesn't work at all. search_form = SearchForm(request.args) is broking it. Also you paginated query twice and used search_form.validate() instead of search_form.validate_on_submit(). Was that intentional? I can give you result of print statements from my previous code, from the original post on this thread, ok? What do you think? Because then pagination and search bar works. Only pagination for search results doesnt. And as I mentioned before, could you send me code samples using pastebin if github doesnt work? It would make our work much more simple

- hijaw39777 3 years, 5 months ago

let say your url is like this your_url?jquery=name so in order to get the jquery value , we need to check that in our request.args like this , value = request.args.get('name','')
and in your paginate URL we can pass like this url_for('url_view_name', jquery='name')

- Vengat 3 years, 5 months ago

Yes, when I change to GET, I have url like that. I think we have to stick to GET method and url_for. So again, I'll try few things again, do some coding and text you tomorrow at same hour as today - 20 CET. But I won't comment here, I'll add new answer to this thread with screenshots. Good night and thanks!

- hijaw39777 3 years, 5 months ago

dont provide on answer thread. you can share your code in pastebin or github link on same thread if doesnt work. Good Night :)

- Vengat 3 years, 5 months ago

No results so far... I think I need some rest and move along with other features for this app. I still got so many to do so maybe I'll find a solution in meantime. I'll be coming back to this issue every free time and I'll be texting you of course in this thread. I tried the GET method, also with url_for and still had no results. If you can try check this thread, could it help us somehow?: https://stackoverflow.com/questions/56847985/pagination-in-search-results-with-python-flask

- hijaw39777 3 years, 5 months ago

I also thought about redirecting user to another route. When the user click search button, it redirect him to "result" page and then display the query with specified keyword from earlier route. I have to figure out passing parameteres between routes, flask session might be good. And what do you think about it?

- hijaw39777 3 years, 5 months ago

I am not sure what exactly the issue in your code. May be we can connect sometime tomorrow for 15 mins to check your code. I did my throat surgery otherwise I would happy to connect more than 15 mins :)

- Vengat 3 years, 5 months ago

Good idea, we could connect for example via discord, skype or something else, maybe in the morning about 9:00 CET? (unfortunately I will have no time in the evening). And tell me what you want me to prepare to show, what exact code and errors for example. It might help us figure out more things. By the way, I wish you everything best about your throat!

- hijaw39777 3 years, 5 months ago

i will send you the zoom link. how about tomorrow 9 AM PST?. Just want to see your code and will debug the code.

- Vengat 3 years, 5 months ago

I won't be at home tomorrow at that hour. What about thursday 9 AM PST? Zoom sounds good to me :)

- hijaw39777 3 years, 5 months ago

yes lets plan on Thursday 9 AM PST. I will send out the zoom link.

- Vengat 3 years, 5 months ago

Hi, I am ready for todays meeting. Are you? :)

- hijaw39777 3 years, 5 months ago

sorry i got pulled into other high ticket meeting. How about 9.30 AM PST? I am just postponing 30 mins.

- Vengat 3 years, 5 months ago

Sure, take your time.

- hijaw39777 3 years, 5 months ago

can you join here https://us04web.zoom.us/j/73039342914?pwd=TFhzSEN5L0lWWU9TZVdoZTNCSkgyQT09?

- Vengat 3 years, 5 months ago

Yes but it's password protected.

- hijaw39777 3 years, 5 months ago

You have to send me a password.

- hijaw39777 3 years, 5 months ago

No it doesnt require password. invite

- Vengat 3 years, 5 months ago

Yes it does, I cannot join until I enter the meeting password.

- hijaw39777 3 years, 5 months ago

i never shared the password with anyone before . its strange. In the URL itself you have pwd. Check the link one more time,
https://us04web.zoom.us/j/73039342914?pwd=TFhzSEN5L0lWWU9TZVdoZTNCSkgyQT09

- Vengat 3 years, 5 months ago

Search bar works and pagination works. BUT! Only when I change page number in url like this: http://127.0.0.1:5000/cve_list?search=jquery&submit=search&page=3. Otherwise it backs to original query like before. So for now, I think the problem is on the redirecting, rendering template site, maybe jinja statements? I don't know but yet but I think we're getting closer and closer right now and we're on a very good way!

- hijaw39777 3 years, 5 months ago

Excellent. It would be great if you edit your answer & share it in this same thread answer section. So that it would help others too and help other techions to take a look on it.

- Vengat 3 years, 5 months ago

you have to use url like {url pagenum filter}

- Vengat 3 years, 5 months ago

What do you mean by that? Can you use a pastebin or something? :-)

- hijaw39777 3 years, 5 months ago

i mean the way you are calling the url when you do pagination should be in proper format like this ?pagenum filtervalue

- Vengat 3 years, 4 months ago

if everything looks good then please close the ticket and update the latest updated code which we have discussed, so that it would helpful to others.

- Vengat 3 years, 4 months ago

Hi arun! I did it! I mean, WE DID IT! Finally solved this problem. I'll update thread soon and write good answer to help others in the future. Thank you so much for you help and kindness! :)

- hijaw39777 3 years, 4 months ago

Excellent. Glad to hear 👍

- Vengat 3 years, 4 months ago

Can you please close the ticket?

- Vengat 3 years, 4 months ago

As we dont get any update on this ticket, I am closing the ticket from my end.

- Vengat 3 years, 4 months ago


Wanna Post Ad?
Reach a large group of audience by posting an ad about your business here. For more details
Drop us a Message

Latest Blogs